Artificial Intelligence Based Cardiac Event Predictor Systems and Methods

ABSTRACT

A method and system for determining cardiac disease risk from electrocardiogram trace data is provided. The method includes receiving electrocardiogram trace data associated with a patient, the electrocardiogram trace data having an electrocardiogram configuration including a plurality of leads. One or more leads of the plurality of leads that are derivable from a combination of other leads of the plurality of leads are identified, and a portion of the electrocardiogram trace data does not include electrocardiogram trace data of the one or more leads. The portion of the electrocardiogram data is provided to a trained machine learning model, to evaluate the portion of the electrocardiogram trace data with respect to one or more cardiac disease states. A risk score reflecting a likelihood of the patient being diagnosed with a cardiac disease state within a predetermined period of time is generated by the trained machine learning model based on the evaluation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/829,356, filed May 31, 2022, which claims the benefit of U.S.provisional application 63/194,923, filed May 28, 2021, U.S. provisionalapplication 63/202,436, filed Jun. 10, 2021, and U.S. provisionalapplication 63/224,850, filed Jul. 22, 2021, and which is acontinuation-in-part of U.S. patent application Ser. No. 17/026,092,filed Sep. 18, 2020, which claims the benefit of U.S. provisionalapplication 62/902,266, filed Sep. 18, 2019, U.S. provisionalapplication 62/924,529, filed Oct. 22, 2019, and U.S. provisionalapplication 63/013,897 filed Apr. 22, 2020. The contents of each ofthese applications is incorporated by reference herein in its entiretyand for all purposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

BACKGROUND OF THE DISCLOSURE

The field of the disclosure is predictive ECG testing and morespecifically a system and process for predicting a future medical orhealth condition using deep learning to associate “current” ECG resultswith future medical conditions.

Medical physicians routinely diagnose patient conditions and prescribesolutions to eliminate or minimize the effects of those conditions. Forinstance, when a patient has a bacterial infection, a physician mayprescribe antibiotics which are known to kill bacteria. In addition,where specific patient conditions are known to commonly be precursors tosubsequent medical events, a physician may prescribe solutions thatmitigate the effects of the subsequent conditions. For instance, in thecase of a patient that is suffering from atrial fibrillation (“AF” or“Afib”); e.g., quivering or irregular heartbeat (arrhythmia) that canlead to blood clots, stroke, heart failure and othercardiovascular-related complications), a physician may prescribe a bloodthinner medication that mitigates the likelihood of subsequent stroke.

In the case of most health conditions, the efficacy (e.g., ultimateability to eliminate or mitigate the condition and/or condition effects)of treatment plans is related to how early the condition is detected.Early detection typically means more treatment options that result ineither a complete/quicker recovery and/or a less severe clinicaloutcome. Thus, for instance, if a physician detects AF immediately afterit starts (or ideally immediately before it begins) as opposed to yearsthereafter, likelihood of treatment success can increase appreciably.This is particularly important for diseases like AF where patients oftenare unaware that they even have this potentially dangerous condition,and they present to the hospital with irreparable damage to the brain(in the form of a stroke) instead of being treated before that damagehappens.

Similarly, in many cases, if a physician can discern a relatively highlikelihood that a currently healthy patient will suffer a specificmedical condition prior to occurrence of that condition, the patient canbe prescribed a treatment plan designed to help avoid the condition inthe future. For example, in the case of AF, if a physician is able todiscern that a patient that does not currently suffer AF has anappreciable risk of AF in the future, that patient can be counseled onways to change his or her lifestyle, or increase monitoring for examplewith a wearable device to detect AF, so as to prevent or reduce thepossibility of future bad outcomes related to AF, such as stroke. Forinstance, it is believed that the likelihood of AF in a patientcurrently with no prior history of AF can be reduced appreciably bylifestyle choices including getting regular physical activity, eating aheart-healthy diet, managing high blood pressure, avoiding excessiveamounts of alcohol and caffeine, not smoking and maintaining a healthyweight and ideally these choices should be selected by anyone who has asubstantial risk of future AF.

The electrocardiogram (ECG) is perhaps the most widely usedcardiovascular diagnostic test in the world, with the vast majority ofpeople undergoing this test at some point in their life. Acquisition ofan electrocardiogram involves any measurement of electrical potentialsat various locations throughout the surface of the body that are used toderive a voltage difference between the two locations. This voltagedifference is then plotted as a function of time, for example afteracquiring approximately 250-500 voltage samples per second. This plot ofvoltage as a function of time forms the basis of an ECG and is referredto as an ECG trace. Since all muscles create electrical voltagedifferences during their normal function, and the heart is essentially alarge muscle, various aspects of heart function can be derived fromthese voltage differences (for example, whether the heart is beatingfast or slow or whether certain parts of the heart are abnormallyenlarged). Thus, analysis of an ECG is used to diagnose and treat manydifferent heart diseases.

ECGs can be acquired using a minimum of 2 body surface potentialrecordings (such that a voltage difference can be calculated from thesubtraction of the two electrical potentials). When only one voltagedifference is acquired typically for a duration of at least 10 seconds,this is known as a “rhythm strip”. One common ECG is the 12-lead ECGwhere voltage differences are acquired in 12 different directions (or“leads”) across the surface of the body. Typically, these are acquiredwhile the patient is not performing physical activity (“at rest”),however, they can also be acquired during strenuous activity (“atstress”). While the resting 12-lead ECG is by far the most commonlyacquired type of ECG, there is no limit to the number of different“leads” that can be acquired for an ECG. Machines that acquire ECGs areubiquitous in current clinical practice and consist of electrodes thatare attached to the surface of a patient's body which are then connectedto multiple wires and a machine which can measure the electricalpotential of each wire. This machine can then calculate the voltagedifferences between the different locations and ultimately generate ECGtraces. The ECG traces are visually examined by a physician to identifyany irregularities. AF is one of many irregularities then can beidentified from ECG traces.

While conventional visual ECG analysis by a trained physician appears towork well for assessing whether a patient currently has AF, conventionalECG analysis does not work well for forecasting likelihood of future AFor other medical events (e.g., heart attacks, stroke, death) that mayresult from future AF.

Population-based screening for AF is challenging. The yearly incidenceof AF in the general population is low with reported incidence rates ofless than 10 per 1000 person years under the age of 70. AF is oftenparoxysmal with many episodes lasting less than 24 hours. Currently, themost common screening strategy is opportunistic pulse palpation,sometimes in conjunction with a 12-lead electrocardiogram (ECG) duringroutine medical visits. This strategy may be appropriate in certainpopulations. However, this strategy may miss many cases of AF.

To this end, even to the trained eye of a physician, there is no way toascertain likelihood of future AF from analyzing an ECG trace that doesnot currently include features consistent with AF. Thus, where aphysician determines that an ECG trace has no evidence of AF, thepatient is simply instructed that he/she does not currently have AFwithout any sense of future AF likelihood or the likelihood of future AFrelated complications.

SUMMARY OF THE DISCLOSURE

In one aspect, the present disclosure provides a method includingreceiving electrocardiogram data associated with a patient and anelectrocardiogram configuration including a plurality of leads and atime interval, the electrocardiogram data including, for each leadincluded in the plurality of leads, voltage data associated with atleast a portion of the time interval, receiving an age value associatedwith the patient, receiving a sex value associated with the patient,providing the age value, the sex value, and at least a portion of theelectrocardiogram data to a trained model, the trained model beingtrained to generate a risk score based on input electrocardiogram dataassociated with the electrocardiogram configuration and supplementaryinformation associated with the patient, receiving a risk scoreindicative of a likelihood the patient will suffer from a conditionwithin a predetermined period of time from when the electrocardiogramdata was generated, and outputting the risk score to at least one of amemory or a display for viewing by a medical practitioner or healthcareadministrator.

The method may further include receiving electronic health record dataassociated with the patient and providing at least a portion of theelectronic health record data to the trained model. The electronichealth record data may include at least one of a blood cholesterolmeasurement, a blood cell count, a blood chemistries lab, a troponinlevel, a natriuretic peptide level, a blood pressure, a heart rate, arespiratory rate, an oxygen saturation, a cardiac ejection fraction, acardiac chamber volume, a heart muscle thickness, a heart valvefunction, a diabetes diagnosis, a chronic kidney disease diagnosis, acongenital heart defect diagnosis, a cancer diagnosis, a procedure, amedication, a referral for cardiac rehabilitation, or a referral fordietary counseling.

The method may further include determining that the risk score is abovea predetermined threshold associated with the condition, in response todetermining that the risk score is above the predetermined threshold,generating a report including information and/or links to sourcesassociated with at least one of treatments for the condition or causesof the condition, and outputting the report to at least one of a memoryor a display for viewing by a medical practitioner or healthcareadministrator.

In the method, the period of time may be one year.

In the method, the period of time may be selected from a range of oneday to thirty years.

In the method, the trained model may include a deep neural networkincluding a plurality of branches. The portion of the electrocardiogramdata provided to the trained model may be provided to the plurality ofbranches.

In the method, the trained model may include a deep neural networkincluding a convolutional component and a dense layer component. Theconvolutional component may include an inception block including aplurality of convolutional layers.

In the method, the plurality of leads may include a lead I, a lead V2, alead V4, a lead V3, a lead V6, a lead II, a lead VI, and a lead V5. Theelectrocardiogram data may include first voltage data associated withthe lead I and a first portion of the time interval, second voltage dataassociated with the lead V2 and a second portion of the time interval,third voltage data associated with the lead V4 and a third portion ofthe time interval, fourth voltage data associated with the lead V3 andthe second portion of the time interval, fifth voltage data associatedwith the lead V6 and the third portion of the time interval, sixthvoltage data associated with the lead II and the first portion of thetime interval, seventh voltage data associated with the lead II and thesecond portion of the time interval, eighth voltage data associated withthe lead II and the third portion of the time interval, ninth voltagedata associated with the lead VI and the first portion of the timeinterval, tenth voltage data associated with the lead VI and the secondportion of the time interval, eleventh voltage data associated with thelead VI and the third portion of the time interval, twelfth voltage dataassociated with the lead V5 and the first portion of the time interval,thirteenth voltage data associated with the lead V5 and the secondportion of the time interval, and fourteenth voltage data associatedwith the lead V5 and the third portion of the time interval. The timeinterval may include a ten second time period, the first portion of thetime interval may include a first half of the time interval, the secondportion of the time interval may include a third quarter of the timeinterval, and the third portion of the time interval may include afourth quarter of the time interval. The trained model may include afirst channel, a second channel, and a third channel, and the providingstep may include providing the first voltage data, the sixth voltagedata, the ninth voltage data, and the twelfth voltage data to the firstchannel, providing the second voltage data, the fourth voltage data, theseventh voltage data, the tenth voltage data, and the thirteenth voltagedata to the second channel, and providing the third voltage data, thefifth voltage data, the eighth voltage data, the eleventh voltage data,and the fourteenth voltage data to the third channel. Each of theplurality of leads may be associated with the time interval.

In the method, the electrocardiogram data may be indicative of a heartcondition based on cardiological standards.

In the method, the electrocardiogram data may not be indicative of aheart condition based on cardiological standards.

In the method, the condition may be mortality.

In the method, the condition may be atrial fibrillation.

In another aspect, the present disclosure provides a method includingreceiving patient electrocardiogram data associated with a patient andan electrocardiogram configuration including a plurality of leads and atime interval from an electrocardiogram device, the patientelectrocardiogram data including, for each lead included in theplurality of leads, voltage data associated with at least a portion ofthe time interval, providing at least a portion of the patientelectrocardiogram data to a trained model, the trained model beingtrained to output a risk score based on input electrocardiogram dataassociated with the electrocardiogram configuration, receiving a riskscore indicative of a likelihood the patient will suffer from acondition within a predetermined period of time from when the patientelectrocardiogram data was generated, generating a report based on therisk score, and outputting the report to at least one of a memory or adisplay for viewing by a medical practitioner or healthcareadministrator.

In yet another aspect, the present disclosure provides a systemincluding at least one processor coupled to at least one memoryincluding instructions. The at least one processor executes theinstructions to receive electrocardiogram data associated with a patientand an electrocardiogram configuration including a plurality of leadsand a time interval, the electrocardiogram data including, for each leadincluded in the plurality of leads, voltage data associated with atleast a portion of the time interval, provide at least a portion of theelectrocardiogram data to a trained model, the trained model beingtrained to output a risk score based on input electrocardiogram dataassociated with the electrocardiogram configuration, receive a riskscore indicative of a likelihood the patient will suffer from acondition within a predetermined period of time from when theelectrocardiogram data was generated from the trained model, and outputthe risk score to at least one of a memory or a display for viewing by amedical practitioner or healthcare administrator.

In still yet another aspect, the present disclosure provides a methodincluding receiving electrocardiogram data associated with a patient andan electrocardiogram configuration including a plurality of leads and atime interval, the electrocardiogram data including, for each leadincluded in the plurality of leads, voltage data associated with atleast a portion of the time interval, receiving demographic dataassociated with the patient, providing the electrocardiogram data andthe demographic data to a trained model, generating information based onthe electrocardiogram data, concatenating the information with thedemographic data, generating a risk score indicative of a likelihood thepatient will suffer from a condition within a predetermined period oftime from when the electrocardiogram data was generated based on theinformation and the demographic data, receiving the risk score from thetrained model, and outputting the risk score to at least one of a memoryor a display for viewing by a medical practitioner or healthcareadministrator.

In the method, the demographic data may include a sex of the patient.

In the method, the demographic data may include an age of the patient.

In the method, the condition may be mortality.

In the method, the condition may be atrial fibrillation.

In the method, the time period may be at least six months. The timeperiod may be at least one year.

In the method, the plurality of leads may include a lead I, a lead V2, alead V4, a lead V3, a lead V6, a lead II, a lead VI, and a lead V5.

The method may further include generating a report based on the riskscore and outputting the report to the display for viewing by a medicalpractitioner or healthcare administrator.

In one aspect, a method includes: receiving electrocardiogram dataassociated with a patient and an electrocardiogram configurationincluding a plurality of leads and a time interval, theelectrocardiogram data comprising, for each lead included in theplurality of leads, voltage data associated with at least a portion ofthe time interval; receiving an age value associated with the patient;receiving a sex value associated with the patient; providing the agevalue, the sex value, and at least a portion of the electrocardiogramdata to a trained model, the trained model being trained to generate arisk score based on input electrocardiogram data associated with theelectrocardiogram configuration and supplementary information associatedwith the patient; receiving a risk score indicative of a likelihood thepatient will suffer from aortic stenosis within a predetermined periodof time from when the electrocardiogram data was generated; andoutputting the risk score to at least one of a memory or a display forviewing by a medical practitioner or healthcare administrator.

Providing the age value, the sex value, and at least a portion of theelectrocardiogram data to a trained model further comprises: providingthe at least a portion of the electrocardiogram data to a convolutionalneural network; and providing the age value and the sex value to aboosting model.

The trained model further comprises: training a convolutional neuralnetwork on a plurality of patients, wherein the plurality of patientsinclude at least patients having a recorded ECG within a diagnosisthreshold and patients having a recorded ECG outside a diagnosisthreshold; wherein the diagnosis threshold is compared against the timebetween the date of diagnosis of aortic stenosis and the date of therecorded ECG; and providing the trained convolutional neural network asthe trained model. The trained model further comprises: refining thetrained neural network using only the plurality of patients having therecorded ECG outside of the diagnosis threshold, wherein the diagnosisthreshold is selected from a number of days.

In another aspect, a method includes: receiving electrocardiogram dataassociated with a wearable device and a patient and an electrocardiogramconfiguration including at least one lead and a time interval, theelectrocardiogram data comprising voltage data associated with at leasta portion of the time interval; receiving an age value associated withthe patient; receiving a sex value associated with the patient;providing the age value, the sex value, and at least a portion of theelectrocardiogram data to a trained model, the trained model beingtrained to generate a risk score based on input electrocardiogram dataassociated with the electrocardiogram configuration and supplementaryinformation associated with the patient; receiving a risk scoreindicative of a likelihood the patient will suffer from aortic stenosiswithin a predetermined period of time from when the electrocardiogramdata was generated; and outputting the risk score to at least one of amemory or a display for viewing by a medical practitioner or healthcareadministrator.

In another aspect, a method includes: receiving electrocardiogram dataassociated with a patient and an electrocardiogram configurationincluding a plurality of leads and a time interval, theelectrocardiogram data comprising, for each lead included in theplurality of leads, voltage data associated with at least a portion ofthe time interval; predicting an interventricular septal thickness(IVSD) value from the electrocardiogram data; receiving an age valueassociated with the patient; receiving a sex value associated with thepatient; providing the age value, the sex value, and the IVSD value to atrained model, the trained model being trained to generate a risk scorebased on input information associated with the patient; receiving a riskscore indicative of a likelihood the patient will suffer from cardiacamyloidosis within a predetermined period of time from when theelectrocardiogram data was generated; and outputting the risk score toat least one of a memory or a display for viewing by a medicalpractitioner or healthcare administrator.

In another aspect, a method includes: receiving electrocardiogram dataassociated with a wearable device and a patient and an electrocardiogramconfiguration including at least one lead and a time interval, theelectrocardiogram data comprising voltage data associated with at leasta portion of the time interval; predicting an interventricular septalthickness (IVSD) value from the electrocardiogram data; receiving an agevalue associated with the patient; receiving a sex value associated withthe patient; providing the age value, the sex value, and the IVSD valueto a trained model, the trained model being trained to generate a riskscore based on input information associated with the patient; receivinga risk score indicative of a likelihood the patient will suffer fromcardiac amyloidosis within a predetermined period of time from when theelectrocardiogram data was generated; and outputting the risk score toat least one of a memory or a display for viewing by a medicalpractitioner or healthcare administrator.

In another aspect, a method includes: receiving electrocardiogram dataassociated with a patient and an electrocardiogram configurationincluding a plurality of leads and a time interval, theelectrocardiogram data comprising, for each lead included in theplurality of leads, voltage data associated with at least a portion ofthe time interval; receiving an age value associated with the patient;receiving a sex value associated with the patient; receiving at leastone diagnostic value associated with the patient; receiving a strokephenotyping value associated with the patient; providing the age value,the sex value, the at least one diagnostic value, the stroke phenotypingvalue, and at least a portion of the electrocardiogram data to a trainedmodel, the trained model being trained to generate a risk score based oninput electrocardiogram data associated with the electrocardiogramconfiguration and supplementary information associated with the patient;receiving a risk score indicative of a likelihood the patient willsuffer from a stroke within a predetermined period of time from when theelectrocardiogram data was generated; and outputting the risk score toat least one of a memory or a display for viewing by a medicalpractitioner or healthcare administrator.

Receiving a stroke phenotyping value further comprises: receiving acategorical indication of stroke onset selected from a recent strokeonset, a recent stroke follow-up, and a history of stroke diagnosis. Inaddition, the at least one diagnostic value is selected from: adiastolic blood pressure, systolic blood pressure, heart rate, heartrhythm, height, weight, race, smoking status, comorbidities, currentmedications, structured echocardiogram measurements, and structured ECGvalues associated with the patient.

In another aspect, a method includes receiving electrocardiogram dataassociated with a wearable device and a patient and an electrocardiogramconfiguration including at least one lead and a time interval, theelectrocardiogram data comprising voltage data associated with at leasta portion of the time interval; receiving an age value associated withthe patient; receiving a sex value associated with the patient;receiving at least one diagnostic value associated with the patient;receiving a stroke phenotyping value associated with the patient;providing the age value, the sex value, the at least one diagnosticvalue, the stroke phenotyping value, and at least a portion of theelectrocardiogram data to a trained model, the trained model beingtrained to generate a risk score based on input electrocardiogram dataassociated with the electrocardiogram configuration and supplementaryinformation associated with the patient; receiving a risk scoreindicative of a likelihood the patient will suffer from a stroke withina predetermined period of time from when the electrocardiogram data wasgenerated; and outputting the risk score to at least one of a memory ora display for viewing by a medical practitioner or healthcareadministrator.

In another aspect, a method includes receiving electrocardiogram dataassociated with a wearable device and a patient and an electrocardiogramconfiguration including at least one lead and a time interval, theelectrocardiogram data comprising voltage data associated with at leasta portion of the time interval, the electrocardiogram data furthercomprising QT interval data; receiving an age value associated with thepatient; receiving a sex value associated with the patient; providingthe age value, the sex value, and at least a portion of theelectrocardiogram data to a trained model, the trained model beingtrained to generate a risk score based on input electrocardiogram dataassociated with the electrocardiogram configuration and supplementaryinformation associated with the patient; receiving a risk scoreindicative of a likelihood the patient will suffer from a cardiac eventwithin a predetermined period of time from when the electrocardiogramdata was generated; and outputting the risk score to at least one of amemory or a display for viewing by a medical practitioner or healthcareadministrator.

The method further includes receiving second electrocardiogram dataassociated with the patient and a second electrocardiogram configurationincluding at least one lead and a time interval, the secondelectrocardiogram data comprising, voltage data associated with at leasta portion of the time interval, the second electrocardiogram datafurther comprising QT interval data, wherein the electrocardiogram datacomprises data taken while the patient is not taking at least one drugand/or has not taken the at least one drug within a specified period oftime prior to the electrocardiogram data being taken, and wherein thesecond electrocardiogram data comprises data taken while the patient istaking the at least one drug. The specified period of time is 90 days.The at least one drug is a drug having known or suspected associationswith prolongation of a corrected QT interval. The condition isprolongation of a corrected QT interval. The trained model employs anartificial intelligence engine including a deep neural network. The deepneural network uses electrocardiogram data and a gradient-boosted treeusing a baseline corrected QT interval with age and sex as additionalinputs.

In another aspect, a method includes: receiving electrocardiogram dataassociated with a wearable device and a patient and an electrocardiogramconfiguration including at least one lead and a time interval, theelectrocardiogram data comprising voltage data associated with at leasta portion of the time interval; receiving an age value associated withthe patient; receiving a sex value associated with the patient;providing the age value, the sex value, and at least a portion of theelectrocardiogram data to a trained model, the trained model beingtrained to generate a risk score based on input electrocardiogram dataassociated with the electrocardiogram configuration and supplementaryinformation associated with the patient; receiving a risk scoreindicative of a likelihood the patient will suffer from AtrialFibrillation within a predetermined period of time from when theelectrocardiogram data was generated; and outputting the risk score toat least one of a memory or a display for viewing by a medicalpractitioner or healthcare administrator.

In another aspect, a method includes: receiving electrocardiogram dataassociated with a wearable device and a patient and an electrocardiogramconfiguration including at least one lead and a time interval, theelectrocardiogram data comprising voltage data associated with at leasta portion of the time interval; receiving supplementary informationassociated with the patient; providing at least a portion of theelectrocardiogram data to a trained model, the trained model beingtrained to generate a risk score based on input electrocardiogram dataassociated with the electrocardiogram configuration and thesupplementary information associated with the patient; receiving a riskscore indicative of a likelihood the patient will suffer from a cardiacevent within a predetermined period of time from when theelectrocardiogram data was generated; and outputting the risk score toat least one of a memory or a display for viewing by a medicalpractitioner or healthcare administrator.

In another aspect, a method includes: receiving electrocardiogram dataassociated with a wearable device and a patient and an electrocardiogramconfiguration including at least one lead and a time interval, theelectrocardiogram data comprising voltage data associated with at leasta portion of the time interval; receiving supplementary informationassociated with the patient; receiving a transformed model, where in atransformed model is based at least in part on a model trained fromelectrocardiogram data having two or more leads which has been refinedwith electrocardiogram data associated with the wearable device andhaving at least one lead; providing at least a portion of theelectrocardiogram data to the transformed model, the transformed modelbeing trained to generate a risk score based on input electrocardiogramdata associated with the electrocardiogram configuration and thesupplementary information associated with the patient; receiving a riskscore indicative of a likelihood the patient will suffer from a cardiacevent within a predetermined period of time from when theelectrocardiogram data was generated; and outputting the risk score toat least one of a memory or a display for viewing by a medicalpractitioner or healthcare administrator.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The file of this patent contains at least one drawing/photographexecuted in color. Copies of this patent with colordrawing(s)/photograph(s) will be provided by the Office upon request andpayment of the necessary fee.

FIG. 1 is an example of a system for automatically predicting an Atrialfibrillation (AF) risk score based on electrocardiogram (ECG) data;

FIG. 2 is an example of hardware that can be used in some embodiments ofthe system of FIG. 1 ;

FIG. 3 is an example of raw ECG voltage input data;

FIG. 4A is an exemplary embodiment of a model;

FIG. 4B is another exemplary embodiment of a model;

FIG. 5A is an exemplary flow of training and testing the model of FIG.4A;

FIG. 5B shows a timeline for ECG selection in accordance with FIG. 5A;

FIG. 6A is a flow including steps employed in identification ofpotentially preventable AF-related strokes among all recorded ischemicstrokes in a stroke registry;

FIG. 6B is a timeline for ECG selection in accordance with FIG. 6A;

FIG. 7A is a bar chart of model performance as mean area under thereceiver operating characteristic;

FIG. 7B is a bar chart of model performance as mean area under theprecision-recall curve;

FIG. 7C is a bar graph of model performance as area under the receiveroperating characteristic;

FIG. 7D is a bar graph of precision-recall curves for the populationwith sufficient data for computation of the CHARGE-AF score;

FIG. 7E is a graph of ROC curves with operating points marked for thethree models;

FIG. 7F is a graph of incidence-free survival curves for the high- andlow-risk groups for the operating point shown in A for a follow-up of 30years;

FIG. 7G is a plot of hazard ratios (HR) with 95% confidence intervals(CI) for the three models in subpopulations defined by age groups, sexand normal or abnormal ECG label;

FIG. 7H is a plot of Kaplan-Meier (KM) incidence-free survival curveswithin the holdout set for males in age groups <50 years, 50-65 yearsand >65 years;

FIG. 7I is a plot of Kaplan-Meier (KM) incidence-free survival curveswithin the holdout set for females in age groups <50 years, 50-65 yearsand >65 years;

FIG. 7J is a plot of KM curves for the model (model M0 trained with ECGtraces, age & sex) predicted low-risk and high-risk groups for new onsetAF for males in age groups <50 years, 50-65 years and >65 years

FIG. 7K is a plot of KM curves for the model predicted low-risk andhigh-risk groups for new onset AF for females in age groups <50 years,50-65 years and >65 years;

FIG. 7L is a plot showing a cumulative distribution of time to AFincidence after ECG in the holdout set of a proof-of-concept model.

FIG. 8A is a graph of receiver operating characteristic curves withchosen operating points;

FIG. 8B is a graph of a Kaplan-Meier curve for predicted low andhigh-risk groups in the normal and abnormal ECG subsets at the operatingpoints in FIG. 8A;

FIG. 9 is a graph of model performance as a function of the definitionof time to incident AF after an ECG;

FIG. 10 is graph of a selection of an operating point on an internalvalidation set in a simulated deployment model;

FIG. 11 is a graph of sensitivity of a model to potentially preventAF-related strokes that developed within 1, 2 and 3 years after ECGgeneration as a function of the percentage of the population targeted ashigh risk to develop incident AF;

FIG. 12 is a graph of percent of all incident AF (within 1 yearpost-ECG) and strokes (within 3 years post-ECG) in the population as afunction of patients below the given age threshold;

FIG. 13 is an exemplary process for generating risk scores using amodel, such as the model in FIG. 4A;

FIG. 14 is a graph illustrating the incidence-free proportion curve forpredicted Afib and predicted no-Afib groups (likelihood threshold=0.5)with the available follow-up;

FIG. 15 is a graph illustrating the top % patients with highest risk andthe positive predictive value across all the operating points of thefuture Afib predictive system;

FIG. 16 is a bar plot of the mortality predicting model or systemperformance to predict 1-year mortality with ECG measures and ECGtraces, with and without age and sex as additional features;

FIG. 17 is a graph illustrating the mean KM curves for predicted aliveand dead groups in normal and abnormal ECG subsets beyond 1-yearpost-ECG;

FIG. 18 is a model architecture for a convolutional neural networkhaving a plurality of branches processing a plurality of channels each;

FIG. 19A is a graph of area under a receiver operating characteristiccurve (AUC) for predicting 1-year all-cause mortality;

FIG. 19B is a bar graph indicating the AUC for various lead locationsderived from 2.5-second or 10-second tracings;

FIG. 20A is a plot of ECG sensitivity vs. specificity;

FIG. 20B is a Kaplan-Meier survival analysis plot of survival proportionvs. time in years at a chose operating point (likelihood threshold=0.5;sensitivity: 0.76; specificity: 0.77);

FIG. 21 is a graph of predicted mortality outcomes by three differentcardiologists before and after seeing model results;

FIG. 22A is a graph of incidence-free proportion vs. time in years;

FIG. 22B is a graph of positive predictive value vs. top percentage riskgroup of a population;

FIG. 23 is a plot of ECG sensitivity vs. specificity for multiple ECGand QTc models;

FIG. 24 displays a block diagram of source data to dataset;

FIG. 25A displays a patient timeline used to label (I) positive ECGs,(II) confirmed negative ECGs, and (III) unconfirmed negative ECGs;

FIG. 25B displays a block diagram for a composite model that shows theclassification pipeline for ECG trace and other EHR data;

FIG. 26 depicts a comparison of AUPRC of a composite model as comparedto a plurality of other individual models;

FIG. 27A displays patient-level retrospective deployment results;

FIG. 27B displays a Sankey plot of retrospective deployment results;

FIG. 28A illustrates a potential configuration of an architecturesupporting a composite model for predicting high-risk patients forcardiac amyloidosis;

FIG. 28B illustrates a second potential configuration of an architecturesupporting a composite model for predicting high-risk patients forcardiac amyloidosis;

FIG. 29 is an example of a method for translating AI algorithms frommulti-lead clinical ECGs to portable and consumer ECGs with fewer leads;

FIG. 30 is a bar chart of model performance as mean area under thereceiver operating characteristic; and

FIG. 31 is a bar chart of model performance as mean area under theprecision-recall curve;

DETAILED DESCRIPTION OF THE DISCLOSURE

The various aspects of the subject disclosure are now described withreference to the drawings, wherein like reference numerals correspond tosimilar elements throughout the several views. It should be understood,however, that the drawings and detailed description hereafter relatingthereto are not intended to limit the claimed subject matter to theparticular form disclosed. Rather, the intention is to cover allmodifications, equivalents, and alternatives falling within the spiritand scope of the claimed subject matter.

In the following detailed description, reference is made to theaccompanying drawings which form a part hereof, and in which is shown byway of illustration, specific embodiments in which the disclosure may bepracticed. These embodiments are described in sufficient detail toenable those of ordinary skill in the art to practice the disclosure. Itshould be understood, however, that the detailed description and thespecific examples, while indicating examples of embodiments of thedisclosure, are given by way of illustration only and not by way oflimitation. From this disclosure, various substitutions, modifications,additions, rearrangements, or combinations thereof within the scope ofthe disclosure may be made and will become apparent to those of ordinaryskill in the art.

In accordance with common practice, the various features illustrated inthe drawings may not be drawn to scale. The illustrations presentedherein are not meant to be actual views of any particular method,device, or system, but are merely idealized representations that areemployed to describe various embodiments of the disclosure. Accordingly,the dimensions of the various features may be arbitrarily expanded orreduced for clarity. In addition, some of the drawings may be simplifiedfor clarity. Thus, the drawings may not depict all of the components ofa given apparatus (e.g., device) or method. In addition, like referencenumerals may be used to denote like features throughout thespecification and figures.

Information and signals described herein may be represented using any ofa variety of different technologies and techniques. For example, data,instructions, commands, information, signals, bits, symbols, and chipsthat may be referenced throughout the above description may berepresented by voltages, currents, electromagnetic waves, magneticfields or particles, optical fields or particles, or any combinationthereof. Some drawings may illustrate signals as a single signal forclarity of presentation and description. It will be understood by aperson of ordinary skill in the art that the signal may represent a busof signals, wherein the bus may have a variety of bit widths and thedisclosure may be implemented on any number of data signals including asingle data signal.

The various illustrative logical blocks, modules, circuits, andalgorithm acts described in connection with embodiments disclosed hereinmay be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,circuits, and acts are described generally in terms of theirfunctionality. Whether such functionality is implemented as hardware orsoftware depends upon the particular application and design constraintsimposed on the overall system. Skilled artisans may implement thedescribed functionality in varying ways for each particular application,but such implementation decisions should not be interpreted as causing adeparture from the scope of the embodiments of the disclosure describedherein.

In addition, it is noted that the embodiments may be described in termsof a process that is depicted as a flowchart, a flow diagram, astructure diagram, or a block diagram. Although a flowchart may describeoperational acts as a sequential process, many of these acts can beperformed in another sequence, in parallel, or substantiallyconcurrently. In addition, the order of the acts may be re-arranged. Aprocess may correspond to a method, a function, a procedure, asubroutine, a subprogram, etc. Furthermore, the methods disclosed hereinmay be implemented in hardware, software, or both. If implemented insoftware, the functions may be stored or transmitted as one or moreinstructions or code on a computer-readable medium. Computer-readablemedia includes both computer storage media and communication mediaincluding any medium that facilitates transfer of a computer programfrom one place to another.

It should be understood that any reference to an element herein using adesignation such as “first,” “second,” and so forth does not limit thequantity or order of those elements, unless such limitation isexplicitly stated. Rather, these designations may be used herein as aconvenient method of distinguishing between two or more elements orinstances of an element. Thus, a reference to first and second elementsdoes not mean that only two elements may be employed there or that thefirst element must precede the second element in some manner. Also,unless stated otherwise a set of elements may comprise one or moreelements.

As used herein, the terms “component,” “system” and the like areintended to refer to a computer-related entity, either hardware, acombination of hardware and software, software, or software inexecution. For example, a component may be, but is not limited to being,a process running on a processor, a processor, an object, an executable,a thread of execution, a program, and/or a computer. By way ofillustration, both an application running on a computer and the computercan be a component. One or more components may reside within a processand/or thread of execution and a component may be localized on onecomputer and/or distributed between two or more computers or processors.

The word “exemplary” is used herein to mean serving as an example,instance, or illustration. Any aspect or design described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other aspects or designs.

Furthermore, the disclosed subject matter may be implemented as asystem, method, apparatus, or article of manufacture using standardprogramming and/or engineering techniques to produce software, firmware,hardware, or any combination thereof to control a computer orprocessor-based device to implement aspects detailed herein. The term“article of manufacture” (or alternatively, “computer program product”)as used herein is intended to encompass a computer program accessiblefrom any computer-readable device, carrier, or media. For example,computer readable media can include but are not limited to magneticstorage devices (e.g., hard disk, floppy disk, magnetic strips . . . ),optical disks (e.g., compact disk (CD), digital versatile disk (DVD) . .. ), smart cards, and flash memory devices (e.g., card, stick).Additionally, it should be appreciated that a carrier wave can beemployed to carry computer-readable electronic data such as those usedin transmitting and receiving electronic mail or in accessing a networksuch as the Internet or a local area network (LAN). Of course, thoseskilled in the art will recognize many modifications may be made to thisconfiguration without departing from the scope or spirit of the claimedsubject matter.

Atrial fibrillation (AF) is associated with substantial morbidity,especially when it goes undetected. If new onset AF can be predictedwith high accuracy, screening methods could be used to find it early.The present disclosure provides a deep neural network that can predictnew onset AF from a resting 12-lead electrocardiogram (ECG). Thepredicted new onset AF may assist medical practitioners (e.g., acardiologist) in preventing AF-related adverse outcomes, such as stroke.

A 12-lead electrocardiogram can include a I Lateral lead (also referredto as a I lead), a II Inferior lead (also referred to as a II lead), aIII Inferior lead (also referred to as a III lead), an aVR lead, an aVLLateral lead (also referred to as an aVL lead), an aVF Inferior lead(also referred to as an aVF lead), a V1 Septal lead (also referred to asa V1 lead), a V2 Septal lead (also referred to as a V2 lead), a V3Anterior lead (also referred to as a V3 lead), a V4 Anterior lead (alsoreferred to as a V4 lead), a V5 Lateral lead (also referred to as a V5lead), and a V6 Lateral lead (also referred to as a V6 lead).

Atrial Fibrillation (AF) is a cardiac rhythm disorder associated withseveral important adverse health outcomes including stroke and heartfailure. In patients with AF and risk factors for thromboembolism, earlyanticoagulation has been shown to be effective at preventing strokes.Unfortunately, AF often goes unrecognized and untreated since it isfrequently asymptomatic or minimally symptomatic. Thus, systems andmethods to screen for and identify undetected AF can assist inpreventing strokes.

Population-based screening for AF is challenging for two primaryreasons. One, the yearly incidence of AF in the general population islow with reported incidence rates of less than 10 per 1000 person yearsunder the age of 70. Two, AF is often “paroxysmal” (the patient goes inand out of AF for periods of time) with many episodes lasting less than24 hours. Currently, the most common screening strategy is opportunisticpulse palpation, sometimes in conjunction with a 12-leadelectrocardiogram during routine medical visits. This has been shown tobe cost-effective in certain populations and is recommended in someguidelines. However, studies of implantable cardiac devices havesuggested that this strategy will miss many cases of AF.

A number of continuous monitoring devices are now available to detectparoxysmal and asymptomatic AF. Patch monitors can be worn for up to14-30 days, implantable loop recorders provide continuous monitoring foras long as 3 years, and wearable monitors, sometimes used in conjunctionwith mobile devices, can be worn indefinitely. Continuous monitoringdevices overcome the problem of paroxysmal AF but must still contendwith the overall low incidence of new onset AF and cost and conveniencelimit their use for widespread population screening.

In the present disclosure, systems and methods to accurately predictcardiac events, including future AF, aortic stenosis (AS), cardiacamyloidosis (CA), and/or stroke (SP) from an ECG, which is a widelyutilized and inexpensive test, are described.

FIG. 1 is an example 100 of a system 100 for automatically predicting anAF, AS, CA, and/or SP risk score based on ECG data (e.g., data from aresting 12-lead ECG). In some embodiments, the system 100 can include acomputing device 104, a secondary computing device 108, and/or a display116. In some embodiments, the system 100 can include an ECG database120, a training data database 124, and/or a trained models database 128.In some embodiments, the computing device 104 can be in communicationwith the secondary computing device 108, the display 116, the ECGdatabase 120, the training data database 124, and/or the trained modelsdatabase 128 over a communication network 112. As shown in FIG. 1 , thecomputing device 104 can receive ECG data, such as 12-lead ECG data, andgenerate an AF, AS, CA, and/or SP risk score based on the ECG data. Insome embodiments, the risk score can indicate a predicted risk of apatient developing the cardiac event within a predetermined time periodfrom when the ECG was taken (e.g., three months, six months, one year,five years, ten years, etc.). In some embodiments, the computing device104 can execute at least a portion of an ECG analysis application 132 toautomatically generate the AF, AS, CA, and/or SP risk score.

The system 100 may generate a risk score to provide physicians with arecommendation to consider additional cardiac monitoring for patientswho are most likely to experience atrial fibrillation, atrial flutter,or another relevant condition within the predetermined time period. Insome examples, the system 100 may be indicated for use in patients aged40 and older without current AF or prior AF history. In some examples,the system 100 may be indicated for use in patients without pre-existingand/or concurrent documentation of AF or other relevant condition. Insome examples, the system 100 may be used by healthcare providers incombination with a patient's medical history and clinical evaluation toinform clinical decision making.

In some embodiments, the ECG data may be indicative or not indicative ofa heart condition based on cardiological standards. For example, the ECGdata may be indicative of a fast heartbeat. The system 100 may predict arisk score indicative that the patient will suffer from the cardiaccondition (e.g., AF) based on ECG data that is not indicative of a givenheart condition (e.g., fast heartbeat). In this way, the system maydetect patients at risk for one or more conditions even when the ECGdata appears “healthy” based on cardiological standards. The system 100may predict a risk score indicative that the patient will suffer fromthe condition (e.g., AF) based on ECG data that is indicative of a heartcondition (e.g., fast heartbeat). In this way, the system 100 may detectpatients at risk for one or more conditions when the ECG data indicatesthe presence of a different condition.

The ECG analysis application 132 can be included in the secondarycomputing device 108 that can be included in the system 100 and/or onthe computing device 104. The computing device 104 can be incommunication with the secondary computing device 108. The computingdevice 104 and/or the secondary computing device 108 may also be incommunication with a display 116 that can be included in the system 100over the communication network 112. In some embodiments, the computingdevice 104 and/or the secondary computing device 108 can cause thedisplay 116 to present one or more AF risk scores and/or reportsgenerated by the ECG analysis application 132.

The communication network 112 can facilitate communication between thecomputing device 104 and the secondary computing device 108. In someembodiments, the communication network 112 can be any suitablecommunication network or combination of communication networks. Forexample, the communication network 112 can include a Wi-Fi network(which can include one or more wireless routers, one or more switches,etc.), a peer-to-peer network (e.g., a Bluetooth network), a cellularnetwork (e.g., a 3G network, a 4G network, a 5G network, etc., complyingwith any suitable standard, such as CDMA, GSM, LTE, LTE Advanced, WiMAX,etc.), a wired network, etc. In some embodiments, the communicationnetwork 112 can be a local area network, a wide area network, a publicnetwork (e.g., the Internet), a private or semi-private network (e.g., acorporate or university intranet), any other suitable type of network,or any suitable combination of networks. Communications links shown inFIG. 1 can each be any suitable communications link or combination ofcommunications links, such as wired links, fiber optic links, Wi-Filinks, Bluetooth links, cellular links, etc.

The ECG database 120 can include a number of ECGs. In some embodiments,the ECGs can include 12-lead ECGs. Each ECG can include a number ofvoltage measurements taken at regular intervals (e.g., at a rate of 250HZ, 500 Hz, 1000 Hz, etc.) over a predetermined time period (e.g., 5seconds, 10 seconds, 15 seconds, 30 seconds, 60 seconds, etc.) for eachlead. In some instances, the number of leads may vary (e.g., from 1-12)and the respective sampling rates and time periods may be different foreach lead. In some embodiments, the ECG can include a single lead. Insome embodiments, the ECG database 120 can include one or more AF riskscores generated by the ECG analysis application 132.

The training data database 124 can include a number of ECGs and clinicaldata. In some embodiments, the clinical data can include outcome data,such as whether or not a patient developed AF in a time period followingthe day that the ECG was taken. Exemplary time periods may include 1month, 2 months, 3 months, 4 months, 5 months, 6 months, 7 months, 8months, 9 months, 10 months, 11 months 12 months, 1 year, 2 years, 3years, 4 years, 5 years, 6 years, 7 years, 8 years, 9 years, or 10years. The ECGs and clinical data can be used for training a model togenerate AF risk scores. In some embodiments, the training data database124 can include multi-lead ECGs taken over a period of time (such as tenseconds) and corresponding clinical data. In some embodiments, thetrained models database 128 can include a number of trained models thatcan receive raw ECGs and output AF risk scores. In other embodiments, adigital image of a lead for an ECG may be used. In some embodiments,trained models 136 can be stored in the computing device 104.

FIG. 2 is an example of hardware that can be used in some embodiments ofthe system 100. The computing device 104 can include a processor 204, adisplay 208, one or more input(s) 212, one or more communicationsystem(s) 216, and a memory 220. The processor 204 can be any suitablehardware processor or combination of processors, such as a centralprocessing unit (“CPU”), a graphics processing unit (“GPU”), etc., whichcan execute a program, which can include the processes described below.

In some embodiments, the display 208 can present a graphical userinterface. In some embodiments, the display 208 can be implemented usingany suitable display devices, such as a computer monitor, a touchscreen,a television, etc. In some embodiments, the input(s) 212 of thecomputing device 104 can include indicators, sensors, actuatablebuttons, a keyboard, a mouse, a graphical user interface, a touch-screendisplay, etc.

In some embodiments, the communication system(s) 216 can include anysuitable hardware, firmware, and/or software for communicating with theother systems, over any suitable communication networks. For example,the communication system 216 can include one or more transceivers, oneor more communication chips and/or chip sets, etc. In a more particularexample, communication system 216 can include hardware, firmware, and/orsoftware that can be used to establish a coaxial connection, a fiberoptic connection, an Ethernet connection, a USB connection, a Wi-Ficonnection, a Bluetooth connection, a cellular connection, etc. In someembodiments, the communication system 216 allows the computing device104 to communicate with the secondary computing device 108.

In some embodiments, the memory 220 can include any suitable storagedevice or devices that can be used to store instructions, values, etc.,that can be used, for example, by the processor 204 to present contentusing display 208, to communicate with the secondary computing device108 via communications system(s) 216, etc. The memory 220 can includeany suitable volatile memory, non-volatile memory, storage, or anysuitable combination thereof. For example, the memory 220 can includeRAM, ROM, EEPROM, one or more flash drives, one or more hard disks, oneor more solid state drives, one or more optical drives, etc. In someembodiments, the memory 220 can have encoded thereon a computer programfor controlling operation of computing device 104 (or secondarycomputing device 108). In such embodiments, the processor 204 canexecute at least a portion of the computer program to present content(e.g., user interfaces, images, graphics, tables, reports, etc.),receive content from the secondary computing device 108, transmitinformation to the secondary computing device 108, etc.

The secondary computing device 108 can include a processor 224, adisplay 228, one or more input(s) 232, one or more communicationsystem(s) 236, and a memory 240. The processor 224 can be any suitablehardware processor or combination of processors, such as a centralprocessing unit (“CPU”), a graphics processing unit (“GPU”), etc., whichcan execute a program, which can include the processes described below.

In some embodiments, the display 228 can present a graphical userinterface. In some embodiments, the display 228 can be implemented usingany suitable display devices, such as a computer monitor, a touchscreen,a television, etc. In some embodiments, the inputs 232 of the secondarycomputing device 108 can include indicators, sensors, actuatablebuttons, a keyboard, a mouse, a graphical user interface, a touch-screendisplay, etc.

In some embodiments, the communication system(s) 236 can include anysuitable hardware, firmware, and/or software for communicating with theother systems, over any suitable communication networks. For example,the communication system 236 can include one or more transceivers, oneor more communication chips and/or chip sets, etc. In a more particularexample, communication system(s) 236 can include hardware, firmware,and/or software that can be used to establish a coaxial connection, afiber optic connection, an Ethernet connection, a USB connection, aWi-Fi connection, a Bluetooth connection, a cellular connection, etc. Insome embodiments, the communication system(s) 236 allows the secondarycomputing device 108 to communicate with the computing device 104.

In some embodiments, the memory 240 can include any suitable storagedevice or devices that can be used to store instructions, values, etc.,that can be used, for example, by the processor 224 to present contentusing display 228, to communicate with the computing device 104 viacommunications system(s) 236, etc. The memory 240 can include anysuitable volatile memory, non-volatile memory, storage, or any suitablecombination thereof. For example, the memory 240 can include RAM, ROM,EEPROM, one or more flash drives, one or more hard disks, one or moresolid state drives, one or more optical drives, etc. In someembodiments, the memory 240 can have encoded thereon a computer programfor controlling operation of secondary computing device 108 (orcomputing device 104). In such embodiments, the processor 224 canexecute at least a portion of the computer program to present content(e.g., user interfaces, images, graphics, tables, reports, etc.),receive content from the computing device 104, transmit information tothe computing device 104, etc.

The display 116 can be a computer display, a television monitor, aprojector, or other suitable displays.

Data Selection and Phenotype Definitions

FIG. 3 is an example of raw ECG voltage input data 300. The ECG voltageinput data includes three distinct, temporally coherent branches afterreducing the data representation from 12 leads to 8 independent leads.Specifically, in the example shown in FIG. 3 , leads aVL, aVF and IIImay not need to be used because they are linear combinations of other,retained leads. Adding these leads may negatively impact the performanceof a model due to overloading of data from certain leads (for example,creating duplicate information) and lead to overfitting. In someembodiments, these leads may boost model performance when they do notrepresent duplicate information. Additionally, lead I was computedbetween the 2.5 and 5 second time interval using Goldberger's equation:−aVR=(I+II)/2. In some embodiments, the data can be acquired at 500 Hz.Data not acquired at 500 Hz (such as studies acquired at 250 Hz or 1000Hz) can be resampled to 500 Hz by linear interpolation or downsampling.In some embodiments, there may be one branch having leads over a full 10seconds, 20 seconds, or 60 seconds of one or more leads. In otherembodiments there may be differing time periods for each branch (e.g.,the first branch may include 0-2.5 seconds, the second branch mayinclude 2.5-6 seconds, and the third branch may include 6-10 seconds).In some embodiments, the number of branches may match the number ofdiffering periods (e.g., there may be 10 branches each receiving asubsequent 1 second lead sampled at 100 Hz, there may be 4 branches eachreceiving a subsequent 2.5 second lead sampled at 500 Hz, etc.). In someembodiments, models may be trained and retained for multiple branch,lead, sampling rate, and/or sampling period structures.

As shown, the raw ECG voltage input data 300 can have a predeterminedECG configuration that defines the leads included in the data and a timeinterval(s) that each lead is sampled, or measured, over. In someembodiments, for the raw ECG voltage input data 300, the ECGconfiguration can include lead I having a time interval of 0-5 seconds,lead V2 having a time interval of 5-7.5 seconds, lead V4 having a timeinterval of 7.5-10 seconds, lead V3 having a time interval of 5-7.5seconds, lead V6 having a time interval of 7.5-10 seconds, lead IIhaving a time interval of 0-10 seconds, lead VI having a time intervalof 0-10 seconds, and lead V5 having a time interval of 0-10 seconds. Theentire ECG voltage input data can have a time interval of 0-10 seconds.Thus, some leads may include data for the entire time interval of theECG voltage input data, and other leads may only include data for asubset of the time interval of the ECG voltage input data.

In some embodiments, the ECG voltage input data 300 can be associatedwith a time interval (e.g., ten seconds). The ECG voltage input data 300can include voltage data generated by leads (e.g., lead I, lead V2, leadV4, lead V3, lead V6, lead II, lead VI, and lead V5). In someembodiments, the raw ECG voltage input data 300 can include voltage datagenerated by the leads over the entire time interval. In someembodiments, the voltage data from certain leads may only be generatedover a portion of the time interval (e.g., the first half of the timeinterval, the third quarter of the time interval, the fourth quarter ofthe time interval) depending on what ECG data is available for thepatient. In some embodiments, a digital image of a raw ECG voltage inputdata may be used and each lead identified from the digital image and acorresponding voltage (e.g., digital voltage data) may be estimated fromanalysis of the digital image.

In some embodiments, the ECG voltage input data 300 can include firstvoltage data 304 associated with the lead I and a first portion of thetime interval, second voltage data 308 associated with the lead V2 and asecond portion of the time interval, third voltage data 312 associatedwith the lead V4 and a third portion of the time interval, fourthvoltage data 316 associated with the lead V3 and the second portion ofthe time interval, fifth voltage data 320 associated with the lead V6and the third portion of the time interval, sixth voltage data 324associated with the lead II and the first portion of the time interval,seventh voltage data 328 associated with the lead II and the secondportion of the time interval, eighth voltage data 332 associated withthe lead II and the third portion of the time interval, ninth voltagedata 336 associated with the lead VI and the first portion of the timeinterval, tenth voltage data 340 associated with the lead VI and thesecond portion of the time interval, eleventh voltage data 344associated with the lead VI and the third portion of the time interval,twelfth voltage data 348 associated with the lead V5 and the firstportion of the time interval, thirteenth voltage data 352 associatedwith the lead V5 and the second portion of the time interval, andfourteenth voltage data 356 associated with the lead V5 and the thirdportion of the time interval. In this way, the voltage data associatedwith the portion(s) of the time interval can be provided to the samechannel(s) of a trained model in order to estimate risk scores for thepatient.

FIG. 4A is an exemplary embodiment of a model 400. Specifically, anarchitecture of the model 400 is shown. Artificial intelligence modelsreferenced herein, including model 700 and model 724 discussed furtherbelow, may be gradient boosting models, random forest models, neuralnetworks (NN), regression models, Naive Bayes models, or machinelearning algorithms (MLA). A MLA or a NN may be trained from a trainingdata set. In an exemplary prediction profile, a training data set mayinclude imaging, pathology, clinical, and/or molecular reports anddetails of a patient, such as those curated from an EHR or geneticsequencing reports. MLAs include supervised algorithms (such asalgorithms where the features/classifications in the data set areannotated) using linear regression, logistic regression, decision trees,classification and regression trees, Naïve Bayes, nearest neighborclustering; unsupervised algorithms (such as algorithms where nofeatures/classification in the data set are annotated) using Apriori,means clustering, principal component analysis, random forest, adaptiveboosting; and semi-supervised algorithms (such as algorithms where anincomplete number of features/classifications in the data set areannotated) using generative approach (such as a mixture of Gaussiandistributions, mixture of multinomial distributions, hidden Markovmodels), low density separation, graph-based approaches (such as mincut,harmonic function, manifold regularization), heuristic approaches, orsupport vector machines. NNs include conditional random fields,convolutional neural networks, attention based neural networks, deeplearning, long short term memory networks, or other neural models wherethe training data set includes a plurality of tumor samples, RNAexpression data for each sample, and pathology reports covering imagingdata for each sample. While MLA and neural networks identify distinctapproaches to machine learning, the terms may be used interchangeablyherein. Thus, a mention of MLA may include a corresponding NN or amention of NN may include a corresponding MLA unless explicitly statedotherwise. Training may include providing optimized datasets, labelingthese traits as they occur in patient records, and training the MLA topredict or classify based on new inputs. Artificial NNs are efficientcomputing models which have shown their strengths in solving hardproblems in artificial intelligence. They have also been shown to beuniversal approximators (can represent a wide variety of functions whengiven appropriate parameters). Some MLA may identify features ofimportance and identify a coefficient, or weight, to them. Thecoefficient may be multiplied with the occurrence frequency of thefeature to generate a score, and once the scores of one or more featuresexceed a threshold, certain classifications may be predicted by the MLA.A coefficient schema may be combined with a rule-based schema togenerate more complicated predictions, such as predictions based uponmultiple features. For example, ten key features may be identifiedacross different classifications. A list of coefficients may exist forthe key features, and a rule set may exist for the classification. Arule set may be based upon the number of occurrences of the feature, thescaled weights of the features, or other qualitative and quantitativeassessments of features encoded in logic known to those of ordinaryskill in the art. In other MLA, features may be organized in a binarytree structure. For example, key features which distinguish between themost classifications may exist as the root of the binary tree and eachsubsequent branch in the tree until a classification may be awardedbased upon reaching a terminal node of the tree. For example, a binarytree may have a root node which tests for a first feature. Theoccurrence or non-occurrence of this feature must exist (the binarydecision), and the logic may traverse the branch which is true for theitem being classified. Additional rules may be based upon thresholds,ranges, or other qualitative and quantitative tests. While supervisedmethods are useful when the training dataset has many known values orannotations, the nature of EMR/EHR documents is that there may not bemany annotations provided. When exploring large amounts of unlabeleddata, unsupervised methods are useful for binning/bucketing instances inthe data set. A single instance of the above models, or two or more suchinstances in combination, may constitute a model for the purposes ofmodels, artificial intelligence, neural networks, or machine learningalgorithms, herein.

In some embodiments, the model 400 can be a deep neural network. In someembodiments, the model 400 can receive the input data shown in FIG. 3 .The input data structure to the model 400 can include a first branch 404including leads I, II, V1, and V5, acquired from time (t)=0 (start ofdata acquisition) to t=5 seconds (e.g., the first voltage data, thesixth voltage data, the ninth voltage data, and the twelfth voltagedata); a second branch 408 including leads V1, V2, V3, II, and V5 fromt=5 to t=7.5 seconds (e.g., the second voltage data, the fourth voltagedata, the seventh voltage data, the tenth voltage data, and thethirteenth voltage data); and a third branch 412 including leads V4, V5,V6, II, and V1 from t=7.5 to t=10 seconds (e.g., the third voltage data,the fifth voltage data, the eighth voltage data, the eleventh voltagedata, and the fourteenth voltage data) as shown in FIG. 3 . Thearrangement of the branches can be designed to account for concurrentmorphology changes throughout the standard clinical acquisition due toarrhythmias and/or premature beats. For example, the model 400 may needto synchronize which voltage information or data is acquired at the samepoint in time in order to understand the data. Because the ECG leads arenot all acquired at the same time, the leads may be aligned todemonstrate to the neural network model which data was collected at thesame time. It is noted that not every lead needs to have voltage dataspanning the entire time interval. This is an advantage of the model400, as some ECGs do not include data for all leads over the entire timeinterval. For example, the model 400 can include ten branches, and canbe trained to generate a risk score based in response to receivingvoltage data spanning subsequent one second periods from ten differentleads. As another example, the model 400 can include four branches, andcan be trained to generate a risk score based in response to receivingvoltage data spanning subsequent 2.5 second periods from four differentleads. Certain organizations such as hospitals may use a standardizedECG configuration (e.g., voltage data spanning subsequent one secondperiods from ten different leads). The model 400 can include anappropriate number of branches and be trained to generate a risk scorefor the standardized ECG configuration. Thus, the model 400 can betailored to whatever ECG configuration is used by a given organization.

In some embodiments, the model 400 can include a convolutional component400A, inception blocks 400B, and a fully connected dense layer component400C. The convolutional component 400A may start with an input for eachbranch followed by a convolutional block. Each convolutional blockincluded in the convolutional component 400A can include a 1Dconvolutional layer, a rectified linear activation (RELU) activationfunction, and a batchnorm layer, in series. Next, this convolutionalblock can be followed by four inception blocks 400B in series, whereeach inception block 400B may include three 1D convolutional blocksconcatenated across the channel axis with decreasing filter windowsizes. Each of the four inception blocks 400B can be connected to a 1Dmax pooling layer, where they are connected to another single 1Dconvolutional block and a final global averaging pool layer. The outputsfor all three branches can be concatenated and fully connected to thedense layer component 400C. The dense layer component 400C can includefour dense layers of 256, 64, 8 and 1 unit(s) with a sigmoid function asthe final layer. All layers in the architecture can enforce kernelconstraints and may not include bias terms. In some embodiments, anAdaGrad optimizer can be used with a learning rate of 1e⁻⁴ 45, a linearlearning rate decay of 1/10 prior to early stopping for efficient modelconvergence, and batch size of 2048. While AdaGrad is presented, otherexamples of algorithms which adaptively update the learning rate of amodel, such as through stochastic gradient descent iterative methodsinclude RMSProp, Adam, and backpropagation learning such as the momentummethod. In some embodiments, the model 400 can be implemented using oneor more machine learning libraries, such as Keras, PyTorch, TernsorFlow,Theano, MXNet, scikit-learn, CUDA, Kubeflow, or MLflow. For example, themodel 700 may be implemented using Keras with a TensorFlow backend inpython and default training parameters were used except where specified.In some embodiments, AdaGrad optimizer can be used with a learning rateof 1e⁻⁴ a linear learning rate decay of 1/10 prior to early stopping forefficient model convergence at patience of three epochs, and batch sizeof 2048. In some embodiments, differing model frameworks, hypertuningparameters, and/or programming languages may be implemented. Thepatience for early stopping was set to 9 epochs. In some embodiments,the model 400 can be trained using NVIDIA DGX1 and DGX2 machines witheight and sixteen V100 GPUs and 32 GB of RAM per GPU, respectively.

In some embodiments, the model 400 can additionally receive electronichealth record (EHR) data points such as demographic data 416, which caninclude age and sex/gender as input features to the network, where sexcan be encoded into binary values for both male and female, and age canbe cast as a continuous numerical value corresponding to the date ofacquisition for each 12-lead resting state ECG. In some embodiments,other representations may be used, such as an age grouping 0-9 years,10-19 years, 20-29 years, or other grouping sizes. In some embodiments,other demographic data such as race, smoking status, height, and/orweight may be included. In some embodiments, the EHR data points caninclude laboratory values, echo measurements, ICD codes, and/or caregaps. The EHR data points (e.g., demographic data, laboratory values,etc.) can be provided to the model 400 at a common location.

The EHR data points (e.g., age and sex) can be fed into a 64-unit hiddenlayer and concatenated with the other branches. In some instances, theseEHR features can be extracted directly from the standard 12-lead ECGreport. In some embodiments, the model 400 can generate ECG informationbased on voltage data from the first branch 404, the second branch 408,and the third branch 412. In some embodiments, the model 400 cangenerate demographic information based on the demographic data 416. Insome embodiments, the demographic information can be generated byinputting age and sex were input into a 64-unit hidden layer. Thedemographic information can be concatenated with the ECG information,and the model 400 can generate a risk score 420 based on the demographicinformation and the ECG information. Concatenating the ECG informationwith the separately generated demographic information can allow themodel 400 to individually disseminate the voltage data from the firstbranch 404, the second branch 408, and the third branch 412, as well asthe demographic data 416, which may improve performance over othermodels that provide the voltage data and the demographic data 416 to themodel at the same channel.

In some embodiments, the model 400 can be included in the trained models136. In some embodiments, the risk score 420 can be indicative of alikelihood the patient will suffer from one or more conditions within apredetermined period of time from when electrocardiogram data (e.g., thevoltage data from the leads) was generated. In some embodiments, thecondition can be AF, mortality, ST-Elevation Myocardial Infarction(STEMI), Acute coronary syndrome (ACS), stroke, or other conditionsindicated herein. For example, in some embodiments, the model 400 can betrained to predict the risk of a patient developing AF in apredetermined time period following the acquisition of an ECG based onthe ECG. In some embodiments, the time period can range from one day tothirty years. For example, the time period may be one day, three months,six months, one year, five years, ten years, and/or thirty years.

FIG. 4B is another exemplary embodiment of a model 424. Specifically,another architecture of the model 400 in FIG. 4A is shown. In someembodiments, the model 424 in FIG. 4B can receive ECG voltage datagenerated over a single time interval.

In some embodiments, the model 424 can be a deep neural network. In someembodiments, such as is shown in FIG. 4B, the model 424 can include asingle branch 432 that can receive ECG voltage input data 428 generatedover a single time interval (e.g., ten seconds). As shown, the model 424can receive ECG voltage input data 428 generated over a time interval often seconds using eight leads. In some embodiments, the ECG voltageinput data 428 can include five thousand data points collected over aperiod of 10 seconds and 8 leads including leads I, II, V1, V2, V3, V4,V5, and V6. The number of data points can vary based on the samplingrate used to sample the leads (e.g., a sampling rate of five hundred Hzwill result in five thousand data points over a time period of tenseconds). The ECG voltage input data 428 can be transformed into ECGwaveforms.

As described above, in some embodiments, the ECG voltage input data 428can be “complete” and contain voltage data from each lead (e.g., lead I,lead V2, lead V4, lead V3, lead V6, lead II, lead VI, and lead V5)generated over the entire time interval. Thus, in some embodiments, thepredetermined ECG configuration can include lead I, lead V2, lead V4,lead V3, lead V6, lead II, lead VI, and lead V5 having time intervals of0-10 seconds. The model 424 can be trained using training data havingthe predetermined ECG configuration including lead I, lead V2, lead V4,lead V3, lead V6, lead II, lead VI, and lead V5 having time intervals of0-10 seconds. When all leads share the same time intervals, the modelcan receive the ECG voltage input data 428 at a single input branch 432.Otherwise, the model can include a branch for each unique time intervalmay be used as described above in conjunction with FIG. 4A.

The ECG waveform data for each ECG lead may be provided to a 1Dconvolutional block 436 where the layer definition parameters (n, f, s)refer, respectively, to the number of data points input presented to theblock, the number of filters used, and the filter size/window. In someembodiments, the number of data points input presented to the block canbe five thousand, the number of filters used can be thirty-two, and thefilter size/window can be eighty. The 1D convolutional block 436 cangenerate and output a downsampled version of the inputted ECG waveformdata to the inception block. In some embodiments, the first 1Dconvolutional block 436 can have a stride value of two.

The model 424 can include an inception block 440. In some embodiments,the inception block 440 can include a number of sub-blocks. Eachsub-block 444 can include a number of convolutional blocks. For example,each sub-block 444 can include a first convolutional block 448A, asecond convolutional block 448B, and a third convolutional block 448C.In the example shown in FIG. 4B, the inception block 440 can includefour sub-blocks in series, such that the output of each sub-block is theinput to the next sub-block. Each inception sub-block can generate andoutput a downsampled set of time-series information. Each sub-block canbe configured with filters and filter windows as shown in the inceptionblock 440 with associated layer definition parameters.

In some embodiments, the first convolutional block 448A, the secondconvolutional block 448B, and the third convolutional block 448C can be1D convolutional blocks. Results from each of the convolutional blocks444A-C can be concatenated 452 by combining the results (e.g., arrays),and inputting the concatenated results to a downsampling layer, such asa MaxPool layer 456 included in the sub-block 444. The MaxPool layer 456can extract positive values for each moving 1D convolutional filterwindow, and allows for another form of regularization, modelgeneralization, and prevent overfitting. After completion of all fourinception block processes, the output is passed to a final convolutionalblock 460 and then a global average pooling (GAP) layer 464. The purposeof the GAP layer 464 is to average the final downsampled ECG featuresfrom all eight independent ECG leads into a single downsampled array.The output of the GAP layer 464 can be passed into the series of denselayer components 424C as in conjunction with FIG. 4A (e.g., at the denselayer component 400C). Furthermore, optimization parameters can also beset for all layers. For example, all layer parameters can enforce akernel constraint parameter (max_norm=3), to prevent overfitting themodel. The first convolutional block 436 and the final convolutionalblock 460 can utilize a stride parameter of n=1, whereas each inceptionblock 440 can utilize a stride parameter of n=2. The stride parametersdetermine the movement of every convolutional layer across the ECG timeseries and can have an impact on model performance. In some embodiments,the model 424 can also concatenate supplementary data such as age andsex as described above in conjunction with FIG. 4A, and the model 424can utilize the same dense layer component architecture as the model400. The model 424 can output a risk score 468 based on the demographicinformation and the ECG information. Specifically, the dense layercomponents 424C can output the risk score 468. In some embodiments, therisk score 420 can be indicative of a likelihood the patient will sufferfrom a condition within a predetermined period of time from whenelectrocardiogram data (e.g., the voltage data from the leads) wasgenerated. In some embodiments, the condition can be AF, mortality,ST-Elevation Myocardial Infarction (STEMI), Acute coronary syndrome(ACS), stroke, or other conditions indicated herein. In someembodiments, for example, the model 400 can be trained to predict therisk of a patient developing AF in a predetermined time period followingthe acquisition of an ECG based on the ECG. In some embodiments, thetime period can range from one day to thirty years. For example, thetime period may be one day, three months, six months, one year, fiveyears, ten years, and/or thirty years.

FIG. 5A is an exemplary flow 500 of training and testing the model 400in FIG. 4A, although it will be appreciated that other training and/ortesting procedures may be implemented. 2.8 million standard 12-lead ECGtraces were extracted from a medical database. All ECGs with knowntime-to-event or minimum 1-year follow-up were used during modeltraining and a single random ECG was selected for each patient in theholdout set for model evaluation, with results denoted as ‘M0’ in FIG.5B. FIG. 5B shows a timeline for ECG selection in accordance with FIG.5A. The traces were acquired between 1984 and June 2019. Additionalretraining was performed only the resting 12-lead ECGs: 1) acquired inpatients ≥18 years of age, 2) with complete voltage-time traces of 2.5seconds for 12 leads and 10 seconds for 3 leads (V1, II, V5), and 3)with no significant artifacts. This amounted to 1.6 million ECGs from431 k patients. The median (interquartile range) follow-up availableafter each ECG was 4.1 (1.5-8.5) years. Each ECG was defined as normalor abnormal as follows: 1) normal ECGs were defined as those withpattern labels of “normal ECG” or “within normal limits” and no otherabnormalities identified; 2) all other ECGs were considered abnormal.Note that a normal ECG does not imply that the patient was free of heartdisease or other medical diagnoses. All the ECG voltage-time traces werepreprocessed to ensure that waveforms were centered around the zerobaseline, while preserving variance and magnitude features.

All studies from patients with pre-existing or concurrent documentationof AF were excluded, it being understood that this process can beadapted to patients with pre-existing or concurrent documentation of oneor more other disease types if the model 700 is being used to evaluateECG data with respect to those disease types in addition to or insteadof AF. Thus, it should be understood that the discussion below can beadapted to those other disease states by substituting those diseasestates for the “AF” references and/or by defining features of thosedisease states. The AF phenotype was defined as a clinically reportedfinding of atrial fibrillation or atrial flutter from a 12-lead ECG or adiagnosis of atrial fibrillation or atrial flutter applied to two ormore inpatient or outpatient encounters or on the patient problem listfrom the institutional electronic health record (EHR) over a 24-yeartime period. Any new diagnoses occurring within 30 days followingcardiac surgery or within one year of a diagnosis of hyperthyroidismwere excluded. Details on the applicable diagnostic codes and blindedchart review validation of the AF phenotype are provided in Table 1below. Atrial flutter was grouped with atrial fibrillation because theclinical consequences of the two rhythms are similar, including the riskof embolization and stroke, and because the two rhythms often coexist.In some embodiments, differing data may be selected for training,validation, and/or test sets of the model.

Table 1 shows performance measures for the blinded chart review of theAF phenotype definition. Diagnostic codes (ICD 9, 10 and EDG) andcorresponding description may be used in defining AF phenotype.

TABLE 1 Blinded chart review validation (AF phenotype) PositivePredictive Value 94.4% Negative Predictive Value  100% Sensitivity  100%Specificity 91.6% True Positive 117  True Negative 76  False Positive 7False Negative 0

AF was considered “new onset” if it occurred at least one day after thebaseline ECG at which time the patient had no history of current orprior AF. EHR data were used to identify the most recent qualifyingencounter date for censorship. Qualifying encounters were restricted toECG, echocardiography, outpatient visit with internal medicine, familymedicine or cardiology, any inpatient encounter, or any surgicalprocedure.

For all experiments, data were divided into training, internalvalidation, and test sets. The composition of the training and test setsvaried by experiment, as described below; however, the internalvalidation set in all cases was defined as a 20% subset of the trainingdata to track validation area under the receiver operatingcharacteristic curve (AUROC) during training to avoid overfitting byearly stopping. The patience for early stopping was set to 9 and thelearning rate was set to decay after 3 epochs when there was noimprovement in the AUROC of the internal validation set during training.

The models were evaluated using the AUROC, which is a robust metric ofmodel performance that represents the ability to discriminate betweentwo classes, although it will be appreciated that other metrics may beused in order to evaluate model performance. Higher AUROC suggestshigher performance (with perfect discrimination represented by an AUROCof 1 and an AUROC of 0.5 being equivalent to a random guess). MultipleAUROCs were compared by bootstrapping 1000 instances (using random andvariable sampling with replacement). Differences between models wereconsidered statistically significant if the absolute difference in the95% CI was greater than zero. The models were also evaluated using areaunder the precision recall curve (AUPRC) as average precision score bycomputing weighted average of precisions achieved at each threshold bythe increase in recall.

Study Design

Two separate modeling experiments were performed as illustrated in FIG.5A.

DNN Prediction Proof-of-Concept (POC)

Using all ECGs from a 15-year period, patients were randomly split intoa training set (DO dataset: 80% of qualifying studies) and a holdouttest set (20%) without overlap of patients between sets. Two versions ofthe model architecture were compared (as described above): one with ECGvoltage versus time traces alone as inputs, and a second with ECG tracesas well as age and sex. Results derived from the holdout test set weredenoted as model ‘M0’. For comparison, a boosted decision-tree basedmodel using only age and sex as inputs and the published CHARGE-AF5-year risk prediction model were implemented in patients with allnecessary data available (requiring age, race, height, weight, systolicand diastolic blood pressure, smoking status, use of antihypertensivemedications, and presence or absence of diabetes, heart failure, andhistory of myocardial infarction. In some embodiments, race and/orsmoking status may not be used. To further evaluate modelgeneralizability, 5-fold cross validation (CV) was performed within theDO dataset to derive models M1-M5. There was no overlap of patientsbetween the train and test sets in each fold. All ECGs with knowntime-to-event or follow-up were used during model training and a singlerandom ECG for a patient was chosen from the test set in all models (M0and M1-M5) so as not to overweight patients with multiple ECGs.

To demonstrate that there was no bias from selecting a single random ECGfrom each patient in the POC model, the performance of the M0 model wasdetermined to be stable without bias across 100 random iterations ofselections with mean and standard deviation of AUROCs and AUPRCs of0.834±0.002 and 0.209±0.004, respectively, for the model with input ofECG traces only; and, 0.845±0.002 and 0.220±0.004 for the model withinput of ECG traces with age and sex.

Kaplan-Meier incidence-free survival analysis was also performed basedon the POC model with the available follow-up data stratified by the DNNmodel prediction, using an optimal operating point to stratify thepopulation into low and high-risk groups. The optimal operating pointfor the M0 model was defined as the point on the ROC curve on thehighest iso-performance line (equal cost to misclassification ofpositives and negatives) in the internal validation set, and thatthreshold was applied to the test set. The data were censored based onthe most recent encounter or development of AF. A Cox ProportionalHazard model regressing time to incidence of AF on the DNNmodel-predicted classification of low-risk and high-risk in the subsetof normal ECGs and the subset of abnormal ECGs was fit. The hazardratios with 95% confidence intervals (CI) were reported for all data andthe normal and abnormal subsets for models M0 and M1-M5 (mean value withlower and upper bounds of 95% CI). The lifelines package (version:0.24.1) in Python was used for survival analysis.

Simulated Deployment Model

To simulate areal-world deployment scenario-using the model to predictincident AF and potentially prevent AF-related strokes-a second modelingapproach was used. All ECGs from a 15-year period were used as atraining set. All ECGs from a five-year period were used as a test set.

To account for potential variability in the clinical implementation ofsuch a model (matching the performance to the scope of availableresources and desired screening characteristics), performance wasevaluated across a range of operating points. An operating point can bethe threshold of the model risk that was used to classify high or lowrisk for developing incident AF. For example, an operating point of 0.7would indicate that model risk scores equal to and above 0.7 areconsidered high risk, and risk scores below 0.7 are low risk. Thus,overall model performance can be measured using AUROC and AUPRC scoresthat aggregate multiple operating point performances into a singlemetric. These points were defined based on maxima of the Fb score (forb=0.15, 0.5, 1, and 2) within the internal validation set. Fb scores arefunctions of precision and recall. A b value of 1 is the harmonic meanof precision and recall (e.g. sensitivity), a value of 2 emphasizesrecall, and values of 0.15 and 0.5 attenuate the influence of recallcorrespondingly. Given the substantial variation in incidence of AF withage, the operating point was varied by age. The ECG with the highestrisk for each patient acquired between the five-year period mentionedabove was selected as the test set.

To link deployment model predictions with potentially preventable strokeevents, an internal registry of patients diagnosed with acute ischemicstroke was used. Through an eight-year period, representing the timeinterval included in this analysis, there were 6,569 patients in thisregistry who were treated for ischemic stroke. This registry was used toidentify patients within the deployment model test set with an ischemicstroke subsequent to the test set ECG. A stroke was consideredpotentially preventable if the following criteria were met: 1) thepatient had at least one ECG prior to the stroke that predicted a highrisk of AF for the given operating point, 2) new onset AF was identifiedbetween 3 days prior to the stroke or up to 365 days after the stroke,and 3) the patient was not on anticoagulation at the time of the stroke.To allow for adequate follow-up, strokes that occurred within 3 years ofthe ECG were included as shown in FIG. 6A. FIG. 6A is a flow 600including steps employed in identification of potentially preventableAF-related strokes among all recorded ischemic strokes in the strokeregistry. FIG. 6B shows a timeline for ECG selection in accordance withFIG. 6A.

Results

The AUROC and AUPRC of the POC DNN models for the prediction of newonset AF within 1 year in the holdout set (M0) were 0.83, 95% CI [0.83,0.84] and 0.21 [0.20, 0.22], respectively, for DNN-ECG and 0.85 [0.84,0.85] and 0.22 [0.21, 0.24], respectively, for DNN-ECG-AS. FIG. 7A is abar chart of model performance as mean area under the receiver operatingcharacteristic. FIG. 7B is a bar chart of model performance as mean areaunder the precision-recall curve. The bars represent the meanperformance across the 5-fold cross-validation with error bars showingstandard deviations. The circle represents the M0 model performance onthe holdout set. The three bars represent model performance for (i)Extreme gradient boosting (XGB) model with age and sex as inputs; (ii)DNN model with ECG voltage-time traces as input and (iii) DNN model withECG voltage-time traces, age and sex as inputs. Within the holdout setthere was sufficient data to calculate CHARGE-AF scores for 65% of thepatients. Within this subset, the DNN-ECG-AS showed superior performance(AUROC=0.84, [0.83, 0.85]; AUPRC=0.20 [0.19, 0.22] compared to theCHARGE-AF score (AUROC=0.79 [0.78, 0.80]; AUPRC=0.12 [0.11, 0.13]. FIG.7C is a bar graph of model performance (proof-of-concept model) as areaunder the receiver operating characteristic, and FIG. 7D is a bar graphof precision-recall curves for the population with sufficient data forcomputation of the CHARGE-AF score. The bars represent the meanperformance across the 5-fold cross-validation with error bars showing95% confidence intervals. The circle represents the M0 model performanceon the holdout set. The three bars represent model performance for (i)Extreme gradient boosting (XGB) model with age and sex as inputs; (ii)DNN model with digital ECG traces as input and (iii) DNN model withdigital ECG traces, age and sex as inputs.

This performance represents a significant improvement compared to theXGBoost model using only age and sex (AUROC=0.78; AUPRC=0.13; p<0.05 fordifference in 95% CI by bootstrapping for both DNN models). Similarly,within the 65% of patients in the holdout test set for whom theCHARGE-AF score could be computed (AUROC=0.78; AUPRC=0.13), the DNNshowed superior performance as well (AUROC=0.79; AUPRC=0.12; see FIG.7B).

The KM curves and HR for the three AF-prediction models in FIGS. 7A-Dare illustrated in FIGS. 7E-G with the operating points marked on thecorresponding ROC curves. Generally, FIGS. 7E-G illustrate receiveroperating characteristic (ROC), incidence-free survival curves andhazard ratios in subpopulations for the following three models evaluatedon the holdout set: (1) age & sex only (the inner dash-dot line); (2)DNN model with ECG traces only (outer dashed line) and (3) DNN modelwith ECG traces, age & sex (solid line) for all ECGs in the holdout set.FIG. 7E illustrates ROC curves with operating points marked for thethree models. FIG. 7F illustrates incidence-free survival curves for thehigh- and low-risk groups for the operating point shown in A for afollow-up of 30 years. FIG. 7G shows a plot of hazard ratios (HR) with95% confidence intervals (CI) for the three models in subpopulationsdefined by age groups, sex and normal or abnormal ECG label. Note thatthere is no HR for Age <50 years for model (1) as there was no subjectclassified as high-risk for new onset AF by the model for thatsubpopulation.

The DNN models showed significant HR of 6.7 [6.4, 7.0] and 7.2 [6.9,7.6] in DNN-ECG and DNN-ECG-AS, respectively. Adjusting for age (inincrements of 10 years) and sex (interactions with sex and model weresignificant) the HR were still significant: 3.7 [3.6, 4.1] and 3.1 [2.7,3.4] in females and males, respectively, for the DNN-ECG model and 3.8[3.6, 4.1] and 2.9 [2.5, 3.4] in females and males, respectively, in theDNN-ECG-AS model in FIG. 7F. For unadjusted comparisons, the DNN modelshad higher HR than the XGBoost model (age and sex) within all subsetsdefined by sex, age groups and ECG type (normal or abnormal).

FIG. 7H shows Kaplan-Meier (KM) incidence-free survival curves withinthe holdout set for males in age groups <50 years, 50-65 years and >65years. FIG. 7I shows Kaplan-Meier (KM) incidence-free survival curveswithin the holdout set for females in age groups <50 years, 50-65 yearsand >65 years.

FIG. 7J shows KM curves for the model (model M0 trained with ECG traces,age & sex) predicted low-risk and high-risk groups for new onset AF formales in age groups <50 years, 50-65 years and >65 years. FIG. 7K showsKM curves for the model predicted low-risk and high-risk groups for newonset AF for females in age groups <50 years, 50-65 years and >65 years.

FIGS. 7H and 7I show the KM curves for age groups <50, 50-65, and >65years in males and females respectively. As expected, in both sexes, thesurvival curves are substantially different in each age group. However,FIGS. 7J and 7K show that in each age group the DNN model retains itsability to discriminate between a high risk and low risk population forthe development of new onset AF for males and females respectively.Specifically, FIGS. 7J and 7K show the incidence of AF that occurs in acohort of patients overtime, where at time zero, no one has AF (100%incidence free), and at time N, shows how many patients had an AFincident. The model shows is sensitive to age as a driving featurebecause older patients typically predict higher incidence of AF overtime than younger patients in the cohort. The superiority of the DNNmodel over age and sex alone is most evident in younger age groups andit is noted that no patient under 58 was predicted as high risk by theXGBoost model.

FIG. 8A is a graph of ROC curves with operating points marked for allthe data (black circle on solid line), the normal ECG subset (blackcircle on dashed line) and the abnormal ECG subset (black circle ondotted line). FIG. 8B is a graph of a KM curve for predicted low andhigh-risk groups in the normal and abnormal ECG subsets at the operatingpoints in FIG. 8A. The shaded area is the 95% confidence interval. Thetable below the graph shows the at-risk population for the given timeintervals in the holdout test set. Moreover, the DNN maintained highperformance even within the subgroup of ECGs clinically reported as‘normal’, as well as the abnormal ECGs (FIG. 7; FIG. 8A). These resultswere observed to be both generalizable and robust based on thecomparable performance of the cross-validation models (M1-M5) to M0, andthe stability of the M0 metrics with repeated iteration of randomsampling within the holdout set. Finally, the model maintained highperformance even in the data subset who developed AF 6 months after ECG(these represent true incident cases, such as potentially paroxysmalcases that manifested quickly from 1 day to 6 months after ECG wereexcluded) with AUROC of 0.83 (FIG. 9 ). FIG. 9 is a graph of modelperformance as a function of the definition of time to incident AF afterthe ECG. The y-axis represents the area under the receiver operatingcharacteristic curve (AUROC) and the x-axis represents differentthresholds for defining incident AF. For example, cases corresponding tothe “2” on the x-axis are those who developed AF at least 2 months afterthe baseline ECG (those developing AF within the first 2 months afterECG were excluded). An AUROC of 0.87 for AF presenting exclusivelybetween 1-31 days following the sinus rhythm ECG was computed,consistent with the findings of others for identification of paroxysmalAF from sinus rhythm.

DNN 1-Year AF Risk Prediction is Associated with Long-Term AF Hazard

Survival free of AF as a function of DNN prediction (low risk vs. highrisk for incident AF) is shown in FIG. 8B. While the proportion ofpatients predicted as high risk, 1 year incidence free AF was high, thehigh-risk prediction was associated with a significant increase inlonger term hazard for AF over the next 3 decades. Specifically, thehazard ratios were 7.2 (95% CI. 6.9-7.56) in all ECGs, 8.2 (7.2-9.3) innormal ECGs, and 6.2 (5.9-6.5) in abnormal ECGs comparing thosepredicted high risk versus low risk for the development of AF within 1year. Furthermore, the median incidence-free survival times of the twogroups identified as low risk and high risk were 13 years and greaterthan 30 years, respectively, for normal ECGs and 10 and 28 years,respectively, for abnormal ECGs.

Prediction of New Onset AF Can Enable Prevention of Future Stroke

In the deployment experiment, the model trained on data prior to 2010and tested on data from 2010-2014 exhibited high performance overall for1-year incident AF prediction, with AUROC and AUPRC of 0.83 and 0.17,respectively. Table 2 summarizes additional model performancecharacteristics at specific operating points dictated by maximal F0.15,F0.5, F1, and F2 scores (with progressively increased emphasis on recalle.g., sensitivity) (FIG. 10 ). FIG. 10 is a graph of the selection ofthe operating point on the internal validation set in the simulateddeployment model using the Fb score or Youden index. These differentpoints resulted in 1, 4, 12 and 20% of the overall population beingflagged as high risk, corresponding with 28, 21, 15 and 12% positivepredictive values and 4, 17, 45 and 62% strokes within 3 years of ECGwere potentially preventable, respectively. In each of these cases, thenumber needed to screen (NNS) to find one new AF case at one year waslow (4-9).

Table 2 is summary of the performance of the model trained with ECGs andage and sex to predict one-year incident atrial fibrillation (AF) in thedeployment scenario for four different operating points defined in theindependent internal validation set.

TABLE 2 Model predicted risk for new onset AF within 1 year of ECG NNSNumber of patients # of % of all to predicted high risk for AF ECGs ECGsfind 1 who developed an AF- flagged flagged new Sensitivity relatedstroke within x Operating high high onset (Recall) Specificity years ofECG (NNS) Point risk risk AF (%) (%) x = 1 x = 2 x = 3 F_(0.5) score7958 4.4 5 26.9 96.4  17  41  65 (468) (194) (122) F₁ score 21831 12.1 752 89.3  51 115 167 (428) (190) (131) F₂ score 37428 20.7 9 68.7 81  69158 231 (542) (237) (162) Youden 50995 28.3 11 77.8 73.5  75 182 269index (680) (280) (190)

Independent of the model, 3,497 patients out of 181,969 (1.9%) wereobserved to have a stroke following an ECG within the deployment testset. Of these, 96, 250 and 375 patients had a stroke within 1, 2 and 3years, respectively, of the ECG and received a diagnosis of new AFbetween −3 and 365 days of the stroke. Of those 96, 250, and 375patients, 84, 229, and 342 were not on an anticoagulant at the time ofthe stroke and represent potentially preventable AF-related strokes(FIG. 6A).

FIG. 11 is a graph of sensitivity of the model to potentially preventAF-related strokes that developed within 1, 2 and 3 years after ECG as afunction of the percentage of the population targeted as high risk todevelop incident AF. Grey dotted lines represent the correspondingoptimal operating thresholds from Table 2. FIG. 11 shows the model'spotential for selecting a high-risk population that can then be screenedfor new onset AF with the goal of stroke prevention. Three conclusionscan be drawn from FIG. 11 . One, the ability to identify potentiallypreventable AF-related strokes is proportional to the ability toidentify new AF. Two, a substantial amount of incident AF can beidentified by screening a relatively small percentage of the population.Three, a variable operating point allows for tradeoffs between precisionand recall that can be tailored to varying priorities.

3,497 patients out of 181,969 (1.9%) with ischemic stroke following anECG within the deployment test set (2010-2014) were observed. Of these,96, 250 and 375 patients had a stroke within 1, 2 and 3 years,respectively, of an ECG and received a new diagnosis of AF within 365days following the stroke. Of those 375 patients, 342 were not on ananticoagulant at the time of the stroke, 31 were on anticoagulantmedications for reasons other than AF, and 2 patients had insufficientrecords to determine if they were being treated with anticoagulants atthe time of the stroke. Hence, these 375 represent a cohort at risk ofAF-related strokes at the time of ECG.

Applying the model (trained on data prior to 2010) to this deploymenttest set, good performance for the prediction of new onset AF at oneyear (AUROC=0.83, AUPRC=0.17) was observed. Using an operating pointdetermined by the F₂ score, the sensitivity was 69%, specificity 81%,and number needed to screen (NNS) to find one case of new onset AF atone year was 9. 62% (231 of 375) of patients who had an AF-relatedstroke within 3 years of an ECG were predicted high risk for new onsetAF (FIG. 11 ). The NNS to identify AF in one patient who developed an AFrelated stroke within 3 years of a high-risk prediction was 162. Table 3is a performance summary of the DNN model (with age and sex) forpredicting one-year new onset AF in a deployment scenario and potentialto identify patients at risk for AF-related stroke within 3 years ofECG. Results are shown based on model predictions using the full testset, as well as specified population subsets with varying demographic,clinical setting, or comorbidity characteristics. Table 3 showsfavorable test characteristics in subgroups defined by age, sex, race,comorbidities, clinical setting and CHA₂DS₂VASc score.

TABLE 3 Number New onset AF within 1 year of ECG predicted highProportion NNS risk for AF who Data of ECGs to find developed an AFflagged 1 new Ss AF related Method/ Data incidence high risk onset(Recall) Sp stroke within 3 Data Subgroup (%) (%) (%) AF (%) (%) years(NNS) Full Test Set F₂ score 100 3.5 21 9 69 81 231 (162) Sex Male 454.1 25 9 70 77 109 (106) Female 55 2.9 17 9 67 84 122 (141) Race White97 3.5 21 9 69 81 227 (162) Black 2.3 1.7 11 13 49 90  3 (156) Others0.8 1.2 11 12 75 90  1 (179) Comorbidities CHD 9 7.8 52 8 84 50  66(129) HF 1.3 18.8 77 4 92 27  17 (109) HT 46.7 4.6 28 9 70 74 162 (146)T2DM 14.4 5.3 33 8 74 69  63 (137) None 49 2.2 13 9 65 88  57 (202)above Patient setting Outpatient 49 2.1 13 13 51 87  63 (189) Emergency26 5.2 26 6 77 77 117 (105) Inpatient 6 7.3 41 7 78 62  20 (232) Unknown18 3.4 27 11 73 75  31 (279) Age groups <50 years 32 0.5 2 15 23 98  2(551) 50-65 33 2.2 12 12 47 89  23 (308) years ≥65 males 15 8.4 54 8 8148  91 (164) ≥65 19 6.7 42 8 76 61 115 (125) females CHA₂DS₂VAS <2 531.4 7 12 43 93  18 (382) c score ≥2 47 5.8 36 8 76 66 213 (143) AF:Atrial Fibrillation/Flutter; NNS: Number needed to screen; CHD: CoronaryHeart Disease; HF: Heart Failure; HT: Hypertension; T2DM: Type IIDiabetes Mellitus; Ss: sensitivity; Sp: Specificity

This disclosure describes a deep neural network that, trained on 12-leadresting ECG data, can predict incident AF within 1 year, in patientswithout a history of AF, with high performance (AUROC=0.85). Moreover,it is demonstrated that this DNN outperformed both a clinical model(CHARGE-AF) and a machine learning model using age and sex within thesame dataset. The superiority of the performance of the model comparedwith the reported performances of other models is noted: CHARGE-AF(AUROC=0.77), ARIC (AUROC=0.78), and Framingham (AUROC=0.78). It is alsonoted that the shorter prediction interval of the model 400 (1 yearcompared to 5-10 years) allows for a more actionable prediction, andthat this prediction retains significant prognostic potential over thenext 3 decades. Finally, by identifying a high-risk population that canbe targeted for screening (e.g. with wearable devices or continuousmonitors), the data demonstrate that a significant proportion ofAF-related strokes can likely be prevented.

Over 25% of all strokes are thought to be due to AF, and ˜20% of strokesdue to AF occur in individuals not previously diagnosed with AF. Areal-world scenario was simulated by applying the model 400 to ECGsacquired over a 5-year period and cross-referencing predicted high riskECGs with future ischemic stroke incidences that were deemed potentiallypreventable (concurrent/subsequent identification of AF and no currentuse of anticoagulation). A range of different model operating pointswere considered based on the expectation that implementation of suchscreening initiatives would differ in scope across different health caresettings. These differences would be reflected in varied preferences fortotal screening numbers vs. proportion of AF identified and number ofstrokes potentially prevented.

At one end of this performance spectrum, in which only the top 1% of thepopulation is identified as high risk, positive predictive valuesapproaching 28% were observed for the detection of 1-year AF (NNS forAF=4). This precision translated to screening volumes (NNS) of 120-361for incident strokes occurring between 0 and 3 years from baseline.However, this lower screening volume was offset by a lower total recall(sensitivity) of preventable strokes (4% for strokes within 3 yearspost-ECG). At the other end of the spectrum in which 21% of thepopulation was identified as high risk for developing AF, thepreventable stroke recall improved substantially (62% for strokes within3 years post-ECG), but at the expense of considerable increases inscreening volume for both AF (NNS=9) and stroke (NNS=162-542 for 3-yearor 1-year incidences, respectively). These numbers for screening volumescompare favorably with other well accepted screening tests includingmammography (NNS 476 to prevent 1 breast cancer death ages 60-69),prostate specific antigen (NNS 1410 to prevent one death from prostatecancer), and cholesterol (NNS 418 to prevent one death fromcardiovascular disease).

The model 400 can be incorporated into routine screening such that everyECG is evaluated and high-risk studies could be flagged for follow-upand surveillance. Such increased surveillance could take many differentforms, including systematic pulse palpation, systematic ECG screening,continuous patch monitors worn once or multiple times, intermittent homescreening with a device such as Kardia mobile, or wearable monitors suchas the Apple Watch. While these methods could be used in isolation toscreen for AF, combination with a DNN predictive model may help toovercome the challenges associated with the overall low incidence of AFin the general population, especially in younger age groups. Age isgenerally thought to be the predominant risk factor in guiding AFscreening strategies, yet in this study 38% of all new AF (within a yearof ECG) and 36% of all potentially preventable strokes (within 3 yearsof ECG) occurred under the age of 70.

FIG. 12 is a graph of percent of all incident AF (within 1-yearpost-ECG) and strokes (within 3 years post-ECG) in the population as afunction of patients below the given age threshold. The model 400 can beused in all patients over the age of 18 and has outperformed a modelthat uses age and sex alone.

The model 400 may detect paroxysmal AF and predicting new onset AF. Thisis in distinction to other techniques that focus solely on theidentification of paroxysmal AF without the ability to predict incidentAF. As noted above, the results indicate that the model 400 is doingboth. One piece of evidence supporting our assertion that the DNN modelcan predict truly new onset AF is the continued separation of the KaplanMeier curves up to thirty years after the index ECG as noted in FIGS.7H-K.

Over 25% of all strokes are thought to be due to AF, and ˜20% of strokesdue to AF occur in individuals not previously diagnosed with AF. Once AFis detected anticoagulation is effective at preventing stroke butscreening for AF is difficult due to the paroxysmal nature of AF and thefact that it is often asymptomatic. Screening strategies involving patchmonitors, wearables, and other devices can be used to detect AF but aremost effective in populations with a high prevalence of AF. Theunderlying goal for developing this prediction model is to identify ahigh-risk population that can then be selected for additional monitoringwith the goal of finding AF prior to a stroke.

A real-world scenario was simulated by applying our model to all ECGsacquired within a large regional health system over a 5-year period bycross-referencing predicted high-risk ECGs with future ischemic strokeincidences that were deemed potentially preventable(concurrent/subsequent identification of AF). It was found that a highproportion (62%) of patients who suffered an AF-related stroke werecorrectly predicted as high risk for AF. The NNS to identify AF in onepatient who later suffered an AF-related stroke was 162. This comparesfavorably with other well accepted screening tests including mammography(NNS 476 to prevent 1 breast cancer death ages 60-69), prostate specificantigen (NNS 1410 to prevent one death from prostate cancer), andcholesterol (NNS 418 to prevent one death from cardiovascular disease).Not all patients with AF are at high risk for stroke and scoring systemssuch as CHA₂DS₂VASc are commonly used to determine the need foranticoagulation. A CHA₂DS₂VASc score of 2 or greater is the cupoint mostcommonly used to start an anticoagulant and Table 3 shows that the modelperforms well within that subgroup with a NNS of 8 to find 1 new case ofAF. Table 3 also shows that 92% of patients predicted high risk for AFwho later suffered an AF-related stroke had a CHA₂DS₂VASc score of 2 orgreater and were potentially eligible for anticoagulation

FIG. 13 is an exemplary process 1300 for generating risk scores using amodel. In some embodiments, the model can be the model 400 in FIG. 4A.In some embodiments, the model can be the model 424 in FIG. 4B. The riskscore can be indicative of whether or not a patient will suffer fromand/or develop a condition within a predetermined time period (e.g., sixmonths, one year, ten years, etc.). In some embodiments, the process1300 can be included in the ECG analysis application 132 in FIG. 1 . Insome embodiments, the process 1300 can be implemented as computerreadable instructions on one or more memories or other non-transitorycomputer readable medium, and executed by one or more processors incommunication with the one or more memories or media. In someembodiments, the process 1300 can be implemented as computer readableinstructions on the memory 220 and/or the memory 240 and executed by theprocessor 204 and/or the processor 224.

At 1304, the process 1300 can receive patient data including ECG data.The ECG data can be associated with the patient. In some embodiments,the ECG data can include the ECG voltage input data 300. In someembodiments, the ECG data can be associated with an electrocardiogramconfiguration including a plurality of leads and a time interval. TheECG data can include, for each lead included in the plurality of leads,voltage data associated with at least a portion of the time interval. Insome embodiments, the ECG data can include first voltage data associatedwith the lead I and a first portion of the time interval, second voltagedata associated with the lead V2 and a second portion of the timeinterval, third voltage data associated with the lead V4 and a thirdportion of the time interval, fourth voltage data associated with thelead V3 and the second portion of the time interval, fifth voltage dataassociated with the lead V6 and the third portion of the time interval,sixth voltage data associated with the lead II and the first portion ofthe time interval, seventh voltage data associated with the lead II andthe second portion of the time interval, eighth voltage data associatedwith the lead II and the third portion of the time interval, ninthvoltage data associated with the lead VI and the first portion of thetime interval, tenth voltage data associated with the lead VI and thesecond portion of the time interval, eleventh voltage data associatedwith the lead VI and the third portion of the time interval, twelfthvoltage data associated with the lead V5 and the first portion of thetime interval, thirteenth voltage data associated with the lead V5 andthe second portion of the time interval, and fourteenth voltage dataassociated with the lead V5 and the third portion of the time interval.

The ECG data can include a first branch (e.g., “branch 1”) includingleads I, II, V1, and V5, acquired from time (t)=0 (start of dataacquisition) to t=5 seconds, a second branch (e.g., “branch 2”)including leads V1, V2, V3, II, and V5 from t=5 to t=7.5 seconds, and athird branch (e.g., “branch 3”) including leads V4, V5, V6, II, and V1from t=7.5 to t=10 seconds as shown in FIG. 3 . In some embodiments theprocess 1300 may also receive demographic data and/or other patientinformation associated with the patient. The demographic data caninclude an age value and a sex value of the patient or additionalvariables (e.g., race, weight, height, smoking status, etc.) for examplefrom the electronic health record. In some embodiments, the process 1300can receive one or more EHR data points. In some embodiments, the EHRdata points can include laboratory values (blood cholesterolmeasurements such as LDL/HDL/total cholesterol, blood counts such ashemoglobin/hematocrit/white blood cell count, blood chemistries such asglucose/sodium/potassium/liver and kidney function labs, and additionalcardiovascular markers such as troponins and natriuretic peptides),vital signs (blood pressures, heart rate, respiratory rate, oxygensaturation), imaging metrics (such as cardiac ejection fractions,cardiac chamber volumes, heart muscle thickness, heart valve function),patient diagnoses (such as diabetes, chronic kidney disease, congenitalheart defects, cancer, etc.), treatments (including procedures,medications, referrals for services such as cardiac rehabilitation,dietary counseling, etc.), echo measurements, ICD codes, and/or caregaps.

In some embodiments, the ECG data can be generated over a single timeinterval (e.g., ten seconds). In some embodiments, the ECG data caninclude the ECG voltage input data 428. In some embodiments, the ECGvoltage input data can include five thousand data points collected overa period of 10 seconds and 8 leads including leads I, II, V1, V2, V3,V4, V5, and V6.

In some embodiments, the ECG data can include leads originally sampledat 500 Hz. In some embodiments, the ECG data can include leadsoriginally sampled at 250 Hz and linearly interpolated to 500 Hz. Insome embodiments, the ECG data can include leads originally sampled at1000 Hz and downsampled to 500 Hz. Thus, a variety of ECG systems and/orsampling settings can be used with the same trained model.

At 1308, the process can provide at least a portion of the patient datato a trained model. In some embodiments, the trained model can be themodel 400. In some embodiments, the process 1308 can provide the ECGdata to the model. In some embodiments, the process 1300 can includeproviding the first voltage data, the sixth voltage data, the ninthvoltage data, and the twelfth voltage data to the first channel,providing the second voltage data, the fourth voltage data, the seventhvoltage data, the tenth voltage data, and the thirteenth voltage data tothe second channel, and providing the third voltage data, the fifthvoltage data, the eighth voltage data, the eleventh voltage data, andthe fourteenth voltage data to the third channel. In some embodiments,the ECG data can include voltage data for all leads over the entire timeinterval, and the process 1300 can include providing the voltage data toa single channel included in the trained model. In some embodiments, theprocess 1308 can provide the ECG data and the demographic data and/orthe EHR data points to the model.

At 1312, the process 1300 can receive a risk score from the model. Insome embodiments, the risk score can be an AF risk score that indicatesa predicted risk of a patient developing AF within a predetermined timeperiod from when the electrocardiogram data was generated. In someembodiments, the predetermined time period can be three months, sixmonths, one year, five years, ten years, thirty years, or any other timeperiod selected from the range of six months to thirty years. In someembodiments, the predetermined time period can be at least three months(e.g., three months, six months, etc.). In some embodiments, thepredetermined time period can be at least six months (e.g., six months,one year, etc.). In some embodiments, the predetermined time period canbe at least one year (e.g., one year, five years, etc.). In someembodiments, the predetermined time period can be at least five years(e.g., five years, ten years, etc.)

At 1316, the process can output the risk score to at least one of amemory (e.g., the memory 220 and/or the memory 240) or a display (e.g.,the display 116, the display 208, and/or the display 228). In someembodiments, the display can be in view of a medical practitioner orhealthcare administrator. In some embodiments, the process 1300 cangenerate and output a report based on the risk score. In someembodiments, the report can include the raw risk score and/or graphicsrelated to the risk score. In some embodiments, the process 1300 candetermine that the risk score is above a predetermined thresholdassociated with the condition (e.g., risk scores above the threshold canbe indicative that the patient will suffer from the conditions withinthe predetermined time period). The process 1300 can then generate thereport based on the determination that the risk score is above apredetermined threshold. In some embodiments, in response to determiningthat the risk score is above the predetermined threshold, the process1300 can generate the report to include information (e.g., text) and/orlinks to sources (e.g., one or more hyperlinks) about treatments for thecondition, causes of the condition, and/or other clinical informationabout the condition. In some embodiments, the process 1300 can generatethe report from intermediate results stored in a standardized format,such as a standardized JavaScript Object Notation (JSON) format. Thestandardized format may also be converted to a different format forpresentation to healthcare providers using format conversion software,such as for conversion into a healthcare providers' electronic healthrecord system. In some embodiments, the process 1300 can generate thereport to include name of the test, patient sex, patient date of birth,patient name, institution/physician name, and/or medical record number.In some embodiments, the process 1300 can generate the report to includean ECG waveform, which may, for instance, be a re-display of theoriginal waveform data produced by the ECG or a re-drawn waveform thatis validated for similarity to the original waveform. In someembodiments, the process 1300 can generate the report to include arecommendation, such as a treatment recommendation or a monitoringrecommendation. For example, the report may include a recommendationthat the patient be subject to additional cardiac monitoring, asignificant step forward in detecting undiagnosed disease. As otherexamples, the report may include one or more recommendations forlifestyle modifications shown to reduce AF or other conditions (e.g.,weight loss, alcohol abstinence, etc.), screen for undiagnosed AF orother condition triggers like sleep apnea, conduct more frequentfollow-up, conduct future ECGs, assess heart rhythm via pulse palpation,or prescribe remote cardiac monitors. Physicians may proceed with any ornone of these actions, or other appropriate patient management strategy,based on information from the device in combination with other symptomsand clinical factors. The process 1300 can then end.

A Deep Neural Network for Predicting Incident Atrial FibrillationDirectly from 12-Lead Electrocardiogram Traces

An example of a neural network trained on clinically acquired ECGs isnow described. From 2.7 million clinically-acquired 12-lead ECGs, 1.1million ECGs without Afib (from 237,060 patients) were extracted.Presence or absence of future incident Afib was determined for each ofthe extracted ECGs via subsequent ECG studies and problem list diagnosesprepared by attending physicians. The prevalence of incident Afib was 7%in the entire population and 3% in the subset of 61,142 patients withECGs clinically interpreted as normal.

A multi-class deep convolutional neural network, using 5-foldcross-validation, was trained to predict 1-year incident Afib (e.g., thetarget output variable) with 15 traces per ECG as input. We assessedmodel performance with area under a receiver operating characteristiccurve (AUC) and performed Cox Proportional Hazard analysis onincidence-free curves of the predicted groups. To additionally evaluatemodel performance in the context of opportunistic population screening,we estimated the positive predictive value (PPV) of the model as afunction of the number of patients with highest model-predicted risk tobe screened.

FIG. 14 is a graph illustrating the incidence-free proportion curve forpredicted Afib and predicted no-Afib groups (likelihood threshold=0.5)with the available follow-up. The mean AUC of the predictive model was0.75±0.02. Unit risk score increase was equivalent to 45% increased oddsof developing AF within a year (Odds Ratio: 1.45 [95% confidenceinterval (CI): 1.15-1.66]). Even in the subset of ECGs interpreted as“normal” (e.g., physician was unable to visually identifyirregularities), the AUC was 0.72±0.02.

FIG. 15 is a graph illustrating the top % patients with highest risk andthe positive predictive value across all the operating points of thefuture Afib predictive system. In the setting of potential populationscreening, the interpretation performance corresponds to a PPV of 0.3for screening the highest 1% at risk.

Deep Neural Networks can Predict 1-Year Mortality Directly from ECGSignal, Even when Clinically Interpreted as Normal

1,775,926 12-lead resting ECGs collected from 397,840 patients over 34years, as well as age, sex and survival status were extracted from asingle medical institution's electronic health records. 15 voltage-time250-500 Hz traces (3 standard “long” 10 sec and 12 “short” 2.5 secacquisitions) were extracted from each ECG along with ‘ECG measures’ (30diagnostic patterns and 9 standard measurements). A deep neural networkwas trained to predict 1-year mortality (e.g., a variable output)directly from the ECG traces. A 5-fold cross-validated model usingdifferent variable inputs and Cox Proportional Hazard survival analysiswere performed on the predicted groups to compare performance. Goodpredictive accuracy was identified within the subset of 297,548 ECGscalled “normal” by the physician. A blinded survey of 3 cardiologistswas performed to determine whether they were capable of seeing featuresindicative of mortality risk within the ECG data.

FIG. 16 is a bar plot of the mortality predicting model or systemperformance to predict 1-year mortality with ECG measures and ECGtraces, with and without age and sex as additional features.

FIG. 17 is a graph illustrating the mean KM curves for predicted aliveand dead groups in normal and abnormal ECG subsets beyond 1-yearpost-ECG.

The model trained with the 15 traces alone yielded an average AUC of0.83, which improved to 0.85 after adding age and sex. This model wassuperior to a separate, non-linear model created from the 39 ECGmeasures (AUC=0.77 and 0.81 without and with age and sex, respectively,p<0.001, see FIG. 16 ). Even within the “normal” ECGs, the modelperformance remained high (AUC=0.84), and the hazard ratio was 6.6(p<0.005) beyond 1-year post-ECG (see FIG. 17 ). In the blinded survey,the patterns captured by the model were not visually apparent tocardiologists, even after being shown labeled true positives (dead) andtrue negatives (alive).

In some embodiments, the trained model can be included in the ECGanalysis application 132, and can be used to predict 1-year mortalityusing a process similar to the process 1300 in FIG. 13 .

Many ECG machines create a “portable document format” (PDF) from thevoltage-time traces which may then be stored in the medical record. Theunderlying voltage data may be extracted from these PDFs by firstconverting the PDF to XML and then parsing the XML file for theunderlying data points which make up each of the voltage-time traces.The XML may also be parsed to determine the patient's age, sex, ninecontinuous numerical measurements output by the ECG machine (QRSduration, QT, QTC, PR interval, ventricular rate, average RR intervaland P, Q and T-wave axes) and thirty categorical ECG patterns,including: a normal, left bundle branch block, incomplete left bundlebranch block, right bundle branch block, incomplete right bundle branchblock, atrial fibrillation, atrial flutter, acute myocardial infarction,left ventricular hypertrophy, premature ventricular contractions,premature atrial contractions, first degree block, second degree block,fascicular block, sinus bradycardia, other bradycardia, sinustachycardia, ventricular tachycardia, supraventricular tachycardia,prolonged QT, pacemaker, ischemia, low QRS voltage,intra-atrioventricular block, prior infarct, nonspecific t-waveabnormality, nonspecific ST wave abnormality, left axis deviation, rightaxis deviation, and an early repolarization which may be diagnosed by aphysician. Example code is presented below in APPENDIX A for convertingfrom PDF to SVG format and from SVG to parsed data points.

Inclusion/Exclusion and Outputs from the Method of Reading the ECG

In some embodiments, a predictive model may be trained using a series ofinput variables, such as the ECG PDF, the variables extracted from thePDF, and the targeted output variables, such as a 1-year mortality rate.During the model training phase, labeled data is provided (in which boththe inputs and outputs are known) to allow the model to learn how bestto predict the output variables. Once the model has been trained, it maybe deployed in a situation where only the input variables are known andthe output may include a prediction target of interest. An exemplarytarget of interest may include a risk of 1-year mortality given thecurrent ECG.

For model training, a series of 12-lead ECG traces may be extracted froman institutional clinical database. Such a database may include over 2.6million traces, such as traces acquired of a period of time, including aperiod of time of months, years, or decades. In an example, the resting12-lead ECGs with voltage-time traces of 2.5 seconds for 12 leads and 10seconds for 3 leads (V1, II, V5) that did not have significant artifactsand were associated with at-least a year of follow-up or death within ayear, may be extracted. Artifacts may include those identified by ECGsoftware at the time of ECG; for example, ECG outputs that include“technically limited”, “motion/baseline artifact”, “Warning:interpretation of this ECG, although attempted, may be adverselyaffected by data quality”, “Acquisition hardware fault prevents reliableanalysis”, “Suggest repeat tracing”, “chest leads probably not wellplaced”, “electrical/somatic/power line interference”, or “DefectiveECG”. Extraction may further include 15 voltage-time traces (three10-second leads and twelve 2.5-second leads). As such, a final datasetmay include 1.8 million ECGs where 51% of them were stored at 500 Hz(Hz=samples per second) and the remaining were stored at 250 Hz. Apreprocessing stage may include resampling the 250 Hz ECGs to 500 Hz bylinear interpolation.

Other Inputs for Consideration, Including Additional Endpoints and EHRData

In instances where additional data may inform the model, extraction mayinclude records from electronic health records having additional patientdata such as patient status (alive/dead) which may be generated bycombining each patient's most recent clinical encounters from the EHRand a regularly-updated death index registry. Patient status is used asan endpoint to determine predictions for 1-year mortality after an ECG,however, additional clinical outcomes may also be predicted, including,but not limited to, mortality at any interval (1, 2, 3 years, etc.);mortality associated with heart disease, cardiovascular disease, suddencardiac death; hospitalization for cardiovascular disease; need forintensive care unit admission for cardiovascular disease; emergencydepartment visit for cardiovascular disease; new onset of an abnormalheart rhythm such as atrial fibrillation; need for a heart transplant;need for an implantable cardiac device such as a pacemaker ordefibrillator; need for mechanical circulatory support such as a leftventricular/right ventricular/biventricular assist device or a totalartificial heart; need for a significant cardiac procedure such aspercutaneous coronary intervention or coronary artery bypassgraft/surgery; new stroke or transient ischemic attack; new acutecoronary syndrome; or new onset of any form of cardiovascular diseasesuch as heart failure; or the likelihood of diagnosis from otherdiseases which may be informed from an ECG.

Moreover, additional variables may be added into a predictive model forpurposes of both improving the prediction accuracy of the endpoints andidentifying treatments which can positively impact the predicted badoutcome. For example, by extracting laboratory values (blood cholesterolmeasurements such as LDL/HDL/total cholesterol, blood counts such ashemoglobin/hematocrit/white blood cell count, blood chemistries such asglucose/sodium/potassium/liver and kidney function labs, and additionalcardiovascular markers such as troponins and natriuretic peptides),vital signs (blood pressures, heart rate, respiratory rate, oxygensaturation), imaging metrics (such as cardiac ejection fractions,cardiac chamber volumes, heart muscle thickness, heart valve function),patient diagnoses (such as diabetes, chronic kidney disease, congenitalheart defects, cancer, etc.) and treatments (including procedures,medications, referrals for services such as cardiac rehabilitation,dietary counseling, etc.), a model's accuracy may be improved. Some ofthese variables are “modifiable” risk factors that can then be used asinputs to the models to demonstrate the benefit of using a particulartherapy. For example, a prediction may identify a patient as a 40%likelihood of developing atrial fibrillation in the next year, however,if the model was able to identify that the patient was taking a betablocker, the predicted risk would drop to 20% based on the increaseddata available to the predictive model. In one example, demographic data416 and patient data 1304 may be supplemented with these additionalvariables, such as the extracted laboratory values or modifiable riskfactors.

Machine learning models for implementing a predictive model may includea convolutional neural network (model architecture illustrated in FIG.18 below) having a plurality of branches processing a plurality ofchannels each. FIG. 18 is a model architecture for a convolutionalneural network having a plurality of branches processing a plurality ofchannels each. As shown, in some embodiments, the model can include fivebranches from which an input of three leads as channels concurrent intime, (Branch 1: [I, II, III]; Branch 2: [aVR, aVL, aVF]; Branch 3: [V1,V2, V3]; Branch 4: [V4, V5, V6] and Branch 5: [V1-long, II-long,V5-long]) may be utilized to generate predictions. In some multi-branchCNNs, each branch can represent the 3 leads as they were acquired at thesame time, or during the same heartbeats. For Branch 5, which caninclude the “long leads,” the leads can be sampled for a duration of 10seconds. For the other four branches, the leads can be sampled for aduration of 2.5 seconds.

In a typical 12-lead ECG, four of these branches of 3 leads are acquiredover a duration of 10 seconds. Concurrently, the “long leads” arerecorded over the entire 10 second duration. To improve robustness ofthe CNN, an architecture may be designed to account for these detailssince abnormal heart rhythms, in particular, cause the traces to changemorphology throughout the standard 10 second clinical acquisition. Atraditional model may miss abnormal heart rhythms which present withmorphology deviations during a longer, 10-second read.

A convolutional block may include a 1-dimensional convolution layerfollowed by batch normalization and rectified linear units (ReLU)activations. In one example, the first four branches and last branch mayinclude 4 and 6 convolutional blocks, respectively, followed by a GlobalAverage Pooling (GAP) layer. The outputs of all the branches may then beconcatenated and connected to a series of dense layers, such as a seriesof six layers, including layers having 256 (with dropout), 128 (withdropout), 64, 32, 8 and 1 unit(s) with a sigmoid function as the finallayer. An Adam optimizer with a learning rate of 1e-5 and batch size of2048 may be computed for each model branch in parallel on a separate GPUfor faster computation. Additional architectures may include (1)replacing the GAP layer with recurrent neural networks such as longshort-term memory and gated recurrent units; (2) changing the number ofconvolutional layers with varying filter sizes in all or number ofbranches in the present architecture or in addition, changing the numberof branches in the architecture; (3) addition of derived signals fromthe time-voltage traces such as power spectral densities to the modeltraining; and (4) addition of tabular or derived features from EHR suchas laboratory values, echo measurements, ICD codes, and/or care gaps inaddition to age and sex. In one example, demographic data 416 andpatient data 1304 may be supplemented with these additional tabular orderived features from the EHR of the subject.

Training Method

The training data may be divided into a plurality of folds with a lastfold set aside as a validation set. An exemplary distribution mayinclude five folds with five percent of the training data set aside as avalidation set. The data may be split such that the same patient is notin both training and testing sets for cross-validation. The outcomes maybe approximately balanced in the validation set. Training timing may bebased upon validation loss which may be evaluated upon each traininginterval. Evaluated loss (binary cross-entropy) on the validation setfor each epoch may be sufficient as a criteria. For example, trainingmay be terminated if the validation loss fails to decrease for 10 epochs(as an early-stopping criteria), and the maximum number of epochs may beset to 500. An exemplary model may be implemented using Keras with aTensorFlow backend in python and default training parameters may beused. In other embodiments, other models, programming languages, andparameters may be used. If all leads are sampled for a single commontime period (e.g., twelve leads sampled from 0-10 seconds), then asingle branch of the abovementioned model may be used. Demographicvariables may be added to the model to boost robustness and improvepredictions. As an example, demographic variables of age and sex may beadded to the model by concatenating with the other branches a 64 hiddenunit layer following. In one example training may be performed on anNVIDIA DGX1 platform with eight V100 GPUs and 32 GB of RAM per GPU.Training, however, may be performed via any computing devices, CPUs,GPUs, FPGAs, ASICs, and the like with variations in duration based uponthe available computer power available at each training device. In onexample, fitting a fold on 5 GPUs and each epoch took approximately 10minutes.

For additional external validation, it may be advantageous to utilizedata acquired at a certain hospital or other provider for training, andthen test the model on all data acquired at the other hospitals/otherproviders. Segmenting training and validation sets by institutionsallows formation of an additional independent validation of modelaccuracy.

Model Operation

Once a model is sufficiently trained, the model may be used to predictone or more status associated with a patient based on the patient's ECG.As such, inputs to the trained model include, at a minimum, an ECG. Themodel's accuracy may be increased, and as such add additional utility(with the capability to recommend treatment changes) by havingadditional clinical variable inputs as described in detail above.

Outputs of the trained model may include the likelihood of a futureadverse outcome (potential outcomes are listed in detail above) andpotential interventions that may be performed to reduce the likelihoodof the adverse outcome. An exemplary intervention that may be suggestedincludes notifying the attending physician that if a patient receives abeta blocker medication, their risk of hospitalization may decrease from10% to 5%.

Generating predictions from these models may include satisfying anobjective to determine the future risk of an adverse clinical outcome,in order to ultimately assist clinicians and patients with earliertreatment and potentially even prevention as a result of the earlierintervention. The duration between the ECG and the ultimate prediction(for example 1 year in the case of predicting 1-year mortality) may varydepending on the clinical outcome of interest and the intervention thatmay ultimately be suggested and/or performed. As references above, themodels may be trained for any relevant time duration after the ECGacquisition, such as a period of time including 1, 2, 3, 4 or 5 years(or more), and for any relevant clinical prediction. Additionally, foreach relevant clinical prediction, an intervention may be similarlysuggested based upon either a model learned correlation, or publicationsof interventions. An example may include predicting that a patient has a40% chance of a-fib in the next year; however, if the patient isprescribed (and takes) a beta blocker, that same patient may insteadhave a reduced, 20% chance of developing a-fib in the next year.Incorporating precision medicine at the earliest stages in treatment,such as when the patient incurs a first ECG, allows treating physiciansto make recommendations that may improve the patient's overall qualityof life and prevent unfavorable outcomes before the patient's healthdeteriorates to the point where they seek advanced medical treatment.Furthermore, by incorporating additional variables above and beyond theECG into the training phase of development, the models will learn howcertain treatments/interventions can positively impact patient outcomes,so as to reduce the chance of the adverse clinical outcome of interest.During the operation phase, the model can ingest the ECG and anyrelevant clinical variable inputs and then output predicted likelihoodof the adverse clinical outcome either without or with certaintreatments/interventions. Even if the patient's current treatments areunknown, the model can make suggestions such as: “If this patienthappens to be diabetic, then their chance of 1-year mortality is reducedby 10% if their blood glucose is adequately controlled according toclinical guidelines.”

Additional Exemplary Model Operations

In one embodiment, a sufficiently trained model may predict likelihoodof Afib and include a further suggestion, based upon the patient'sheight, weight, or BMI, that weight loss is needed to improve thepatient's overall response to therapy. A sufficiently trained model mayinclude a model that ingests a PDF of a clinically-acquired 12-leadresting ECG and outputs the precise risk of mortality at 1 year as alikelihood ranging from 0 to 1 where the model also received a patientheight, weight, or BMI and the patient's clinical updates over thecourse of at least a year.

FIG. 19A is a graph of area under a receiver operating characteristiccurve (AUC) for predicting 1-year all-cause mortality. FIG. 19B is a bargraph indicating the AUC for various lead locations derived from2.5-second or 10-second tracings.

Using the inclusion/exclusion criteria described above and a 5-foldcross-validation scheme, it may be demonstrated that the area under thereceiver operating characteristic curve (AUC) for predicting 1-yearall-cause mortality is 0.830 using the ECG voltage-time traces alone(taken directly from the PDF) and improved to 0.847 when age and sexwere added as additional input variables (see the far-right bars in eachgrouping in FIG. 19A). Note that AUC is a measure of model accuracy thatranges from 0.5 (worst predictive accuracy equivalent to random chance)to 1 (perfect prediction). During a 12-lead ECG acquisition, all leadsare acquired for a duration of 2.5 seconds and three of those 12-leads(V1, II and V5) are additionally acquired for a duration of 10 seconds.The model with all 15 ECG voltage-time traces from the 12 standard leadstogether (3 leads acquired for 2.5 seconds plus 12-leads acquired for 10seconds) provided the best AUC compared to models derived from eachsingle lead as input. Models derived from the 10-second tracings hadhigher AUCs than the models derived from the 2.5-second tracings,demonstrating that a longer duration of data provides more informativefeatures to the model.

FIG. 20A is a plot of ECG sensitivity vs. specificity. FIG. 20B is aKaplan-Meier survival analysis plot of survival proportion vs. time inyears at a chose operating point (likelihood threshold=0.5; sensitivity:0.76; specificity: 0.77);

To further investigate predictive performance within the overall datasetand the subsets of ECGs interpreted as either “normal” or “abnormal” bya physician, Kaplan-Meier survival analysis was performed usingfollow-up data available in the EHR for the two groups predicted by themodel (alive/dead in 1-year) at the chosen operating point (likelihoodthreshold=0.5; sensitivity: 0.76; specificity: 0.77). For normal ECGs,the median survival times (for the mean survival curves of five-folds)of the two groups predicted alive and dead at 1-year were 26 and 8years, respectively, and for abnormal ECGs, 16 and 6 years, respectively(see FIG. 20B). A Cox Proportional Hazard regression model was fit foreach of the five folds and mean hazard ratios (with lower and upperbounds of 95% confidence intervals) were: 4.4 [4.0-4.5] in all ECGs, 3.9[3.6-4.0] in abnormal ECGs and 6.6 [5.8-7.6] in normal ECGs (allp<0.005) comparing those predicted by the model to be alive versus deadat 1-year post-ECG. Thus, the hazard ratio was largest in the subset ofnormal ECGs, and the prediction of 1-year mortality from the model was asignificant discriminator of long-term survival for 30 years after theclinical acquisition of the ECG.

FIG. 21 is a graph of predicted mortality outcomes by three differentcardiologists before and after seeing model results. Anotherconsideration of a sufficiently trained model may include if thefeatures learned by the model are visually apparent to cardiologists.For example, if four hundred and one sets of paired normal ECGs areselected and provided to a blinded survey with three cardiologists, ameasure of model performance against cardiologist visual inspection maybe generated. Each pair may consist of a true positive (normal ECGcorrectly predicted by the model as dead at one year) and a truenegative (normal ECG correctly predicted by the model as alive at oneyear), matched for age and sex. FIG. 22A is a graph of incidence-freeproportion vs. time in years. FIG. 22B is a graph of positive predictivevalue vs. top percentage risk group of a population. In one studycardiologists generally had poor accuracy of 55-68% (10-36% above randomchance) to correctly identify the normal ECG linked to 1-year mortality.After allowing each cardiologist to study a separate dataset of 240paired ECGs labeled to show the outcome, their prediction accuracy inrepeating the original blinded survey of 401 paired ECGs remained low(50-75% accuracy or 0-50% above random chance) (see FIG. 21 ). Thissuggests that the above models are able to identify features predictiveof important clinical outcomes that, importantly, cardiologists are notable to visually identify despite many years of clinical training.

Note that the reported accuracies for predicting outcomes can likely beslightly improved by testing against only a single ECG from eachpatient. The above numbers report test data accuracies (AUCs) from allECGs from a patient, which ends up over-weighting patients who receivemore ECGs (patients who receive 20 ECGs in a lifetime contribute more tothe assessment of accuracy than a patient who only received 1 ECG inhis/her lifetime). Since patients who have more ECGs are typicallysicker, it is more difficult to predict their clinical outcomes and thusover-weighting those patients can slightly reduce the perceivedaccuracies (AUCs).

Prediction of Atrial Fibrillation

Atrial fibrillation (AF) is an abnormal rhythm in the heart thatincreases the risk of stroke. Predictive strategies for detecting theonset of AF, before stroke occurs, are therefore highly clinicallyimportant. In one embodiment, a deep learning model may predict futureAF directly from 12-lead resting electrocardiogram (ECG) voltage-timetraces as extracted from a clinically-acquired PDF.

For example, a dataset including 2.7 million clinically-acquired 12-leadECGs, may include 1.1 million ECGs without AF (from 237,060 patients).The presence or absence of future incident AF may be determined viasubsequent ECG studies and problem list diagnoses in the electronichealth record. The prevalence of incident AF was 7% in the entirepopulation and 3% in a subset of 61,142 patients with ECGs clinicallyinterpreted as normal. A model, such as a multi-class deep convolutionalneural network using 5-fold cross-validation, may be trained to predict1-year incident AF with 15 ECG traces as input. In one instance, modelperformance may be measured from the area under the receiver operatingcharacteristic curve (AUC) and Cox Proportional Hazard analysis onincidence-free curves of the predicted groups. Additional evaluation ofmodel performance may be performed in the context of opportunisticpopulation screening. For example, the positive predictive value (PPV)of the model as a function of the number of patients with highestmodel-predicted risk to be screened may be calculated. In themulti-class deep CNN with 15 ECG traces as input instance, the mean AUCof the predictive model was 0.75 and patients predicted to develop AFwithin the next year had a significant long-term increased risk fordeveloping AF that extended over 25 years after the ECG acquisition (seeFIG. 22A). Even in the subset of ECGs interpreted as ‘normal’ by aphysician, the AUC was 0.720. In the setting of potential populationscreening, this performance corresponded to a positive predictive valueof 0.3 for screening the highest 1% at risk (see FIG. 22B). This meansthat, of the top 1% at risk, approximately 30% will end up developing AFwithin the first year, and many more will develop AF over the next 25years.

In summary, this is another example of using a model to predict theonset of a future clinically relevant event (atrial fibrillation withinthe next year). This prediction maintains modest accuracy even when theECG is clinically interpreted as ‘normal’ by a physician. Providingpredictions to the physician, especially in instances where thephysician's ‘normal’ clinical interpretation of the ECG occurs, willgreatly improve patient care. The predictive and therapeuticimplications of the model may be even further improved with theinclusion of additional features to the training phase of the modeldevelopment, allowing even further relevant predictions about howtreatments/interventions reduce the risk of developing AF (for example,if a patient is taking a beta-blocker medication or has his/her bloodpressure within a normal range it will likely reduce the risk ofdeveloping AF, and the model can make these predictions) may be includedin a patient's treatment.

In some embodiments, the results reported by model 400 reflect detectionof paroxysmal AF and prediction of incident AF. Intuitively, thecharacteristics of the ECG that lead to a high-risk prediction by theDNN will be more prevalent in patients who already have AF but arecurrently in sinus rhythm. With this in mind a higher model performancefor identification of paroxysmal AF compared to prediction of incidentAF was expected, and this is exactly what was seen. A declining rate ofnew onset AF over the course of one year also was expected. This is seenin FIG. 7L and is consistent with rapid identification of paroxysmal AFfollowed by a slower identification of cases that represent incident AF.The largest piece of evidence supporting the assertion that the DNNmodel can predict incident AF is the continued separation of the KMincidence-free survival curves up to thirty years after the index ECG asnoted in FIGS. 7E through 7K. In other embodiments, the results frommodel 400 may reflect structural changes that occur in the atria ofpatients with AF, such that the model 400 uses ECG manifestations ofthis atrial myopathy to guide the predictive results it provides.

There are many different settings in which the system 100 may beutilized and the methods disclosed herein may be performed. With regardto setting, one promising opportunity—particularly for integrated caredelivery systems—is the systematic screening of all ECGs in a healthsystem. For example, the model 400 could be incorporated into anexisting clinical workflow (such as through an EHR system) such thatevery ECG is evaluated, and high-risk studies could be flagged forfollow-up and surveillance. Such increased surveillance could take manydifferent forms, including systematic pulse palpation, systematic ECGscreening, continuous patch monitors worn once or multiple times,intermittent home screening with a device such as Kardia mobile, orwearable monitors such as the Apple Watch.

APPENDIX A CODE: (Method of reading ECG) def convert_pdf_to_svg(fname,outname, verbose=0): ''' Input:   fname : PDF file name   outname : SVGfile name Output:   outname : return outname (file saved to disk) Thiswill convert PDF into SVG format and save it in the given outpath. '''(status, out) = subprocess.getstatusoutput(''.join(['pdftocairo -svg ',fname,' ', outname])) if (status != 0):  logging.error('Error inconverting PDF to SVG: { }'.format(out)) return outname defprocess_svg_to_pd_perdata(svgfile, pdffile=None): ''' Input: svgfile -datapath for svg file Output (returns): data : data for 12leads(available 15 or 12 traces), scale vales and resolution units in apandas dataframe Hard coded values : 1) length of signal = 6 is assumedto be the calibration tracing at the beginning of the trace (byexperiment) ''' columnnames = np.array(['I','II','III','aVR','aVL','aVF','V1','V2','V3','V4', \    'V5', 'V6','V1L','IIL','V5L']) doc = parse(svgfile) if pdffile is None: strn =os.path.splitext(os.path.basename(svgfile))[0] else: strn =os.path.splitext(os.path.basename(pdffile))[0] arrayindex =[np.array([strn, strn]), np.array(['x','y'])] data =pd.DataFrame(columns = ['PT_MRN','TEST_ID','filename','lead', 'x','y'])#,'scale_x','scale_y']) a = 0 spacingvals = [ ] scale_vals = [ ] try: siglen = [ ]  for path in doc.getElementsByTagName('path'):    tmp =path.getAttribute('d')   tmp_split = tmp.split(' ')   signal_np =np.asarray([float(x) for x in tmp_split if (x != 'M' and x != 'L' and x!= 'C' and x != 'Z' and x != ' ')])   signalx = signal_np[0::2]  signaly = signal_np[l::2]   siglen.append(len(signalx))  siglen =np.array(siglen)  # these are the calibration signals  cali6sigs =np.where(siglen == 6)[0]  minposcali = np.min(cali6sigs)   tmpstart =list(range(minposcali, len(siglen)))  last15sigs =np.array(list(set(tmpstart)- set(cali6sigs)))  # index for leads  a = 0 for ind, path in enumerate(doc.getElementsByTagName('path')):   if indin last15sigs:    if a > 14:     continue    tmp =path.getAttribute('d')    tmp_split = tmp.split(' ')    signal_np =np.asarray([float(x) for x in tmp_split if (x != 'M' and x != 'L' and x!= 'C' and x != 'Z' andx !='')])    signalx = signal_np[0::2]    signaly= signal_np[l::2]    # expect the name of the file to be ptmrn_testidformat.    tmp = strn.split('_')    try:     pid, testid = tmp[0],tmp[1]    except:     pid = tmp[0]     testid = tmp[0]   data.loc[data.shape[0]] = [pid, testid, strn, columnnames [a],signalx, signaly]    spacingx = [t -s for s,t in zip(signalx,signalx[1:])]     spacingvals.append(np.min(spacingx))    a += 1   elifind in cali6sigs:    tmp = path.getAttribute('d')    tmp_split =tmp.split(' ')    signal_np = np.asarray([float(x) for x in tmp_split if(x != 'M' and x != 'L' and x != 'C' and x != 'Z' and x ='')    signalx =signal_np[0::2]    signaly = signal_np[1::2]   scale_vals.append([np.min(signaly), np.max(signaly)])  iflen(scale_vals) == 0:   data = None return data  sx = [x[0] for x inscale_vals]  sy = [x[1] for x in scale_vals]  startloc = [d[0] for d indata.x.values]  leads_ip = len(startloc)  a = np.sum(startloc[0:3] ==startloc[0])  b = np.sum(startloc[3:6] == startloc[3])  c =np.sum(startloc[6:9] == startloc[6])  d = np.sum(startloc[9:12] ==startloc[9])  if data.shape [0] == 15:   e = np.sum(startloc[12:15] ==startloc[12])   checkrhs = [3,3,3,3,3]   checklhs = [a,b,c,d,e]   assertchecklhs == checkrhs   scale_x= [sx[0:3],sx[0:3],sx[0:3],sx[0:3],sx[3:6]]   scale_y = [sy[0:3],sy[0:3],sy[0:3],sy[0:3], sy[3:6]]  elifdata.shape[0] == 12:   checkrhs = [3,3,3,3]   checklhs = [a,b,c,d]  assert checklhs == checkrhs   scale_x =[sx[0:3],sx[0:3],sx[0:3],sx[0:3]]   scale_y =[sy[0:3],sy[0:3],sy[0:3],sy[0:3]]  else:   data=None   return data scale_x = [y for x in scale x for y in x]  data['scale_x'] =scale_x[0:data.shape[0]]  scale_y = [y for x in scale_y for y in x] data['scale_y'] = scale_y[0: data.shape[0]]  data['minspacing'] =spacingvals[0:data.shape[0]] except:  data = None return data

Thus, a properly trained deep neural network can predict incident AFdirectly from 12-lead ECG traces, even when the ECG is clinicallyinterpreted as “normal”. This approach has significant potential fortargeted screening and monitoring of new onset AF to potentiallyminimize the risk of stroke.

In addition, deep learning can be a powerful tool for identifyingpatients with potential adverse outcomes (e.g., death) who may benefitfrom early interventions, even in cases interpreted as “normal” byphysicians.

In one embodiment, systems and methods described herein for predictionof atrial fibrillation from an ECG may further be adapted to predictother cardiac events from received ECG data. For example, of thereceived ECG data, measurements may record abnormal variations which aremeaningful in additional cardiac event analytics. The QT interval is onesuch measurement made on an ECG used to assess some of the electricalproperties of the heart. It is calculated as the time from the start ofthe Q wave to the end of the T wave, and approximates to the time takenfrom when the cardiac ventricles start to contract to when they finishrelaxing. An abnormally long or abnormally short QT interval isassociated with an increased risk of developing abnormal heart rhythmsand sudden cardiac death. Abnormalities in the QT interval can be causedby genetic conditions such as long QT syndrome, by certain medicationssuch as sotalol or pitolisant, by disturbances in the concentrations ofcertain salts within the blood such as hypokalaemia, by hormonalimbalances such as hypothyroidism, or they may be induced by certainmedications. QT prolongation is a measure of delayed ventricularrepolarization. Excessive QT prolongation can predispose the myocardiumto the development of early after-depolarisations, which in turn cantrigger re-entrant tachycardias such as torsades de pointes (TdP).Although the relationship between QT interval duration and the risk ofTdP is not fully understood, a corrected QT interval (QTc) of >500 ms oran increase in the QTc of >60 ms may be considered to confer a high riskof TdP in an individual patient. Prolongation of the corrected QT (QTc)interval becomes an even further concern, for example, with patients whoreceive psychotropic medications. Such patients may have baselineclinical risk factors for QTc prolongation, and many psychotropicmedications may further prolong this interval. Analytics may identifyover 200 medications having known or suspected association with QTcprolongation (LQT), which can lead to the rare but potentiallycatastrophic event, TdP.

Models herein generate predictions based upon the combination of ECGdata, patient age, and patient sex, although it will be appreciated thatmodels may be generated by combining ECG data with other or additionaldemographic data or EHR-derived patient data. Prediction of drug-inducedLQT using an ECG-based machine learning model is feasible and mayoutperform a model trained on baseline QTc, age, and sex alone. In oneexample, ECG inputs having a baseline 12-lead ECGs with QTc values <500ms for patients who had not received any known, conditional, or possibleQTc prolonging medication at the time of ECG or within the past 90 daysmay be matched with ECGs from the same patients while they were takingat least one drug (“on-drug” ECGs), such as one of the over 200medications having known or suspected associations with LQT. Featuresfrom the ECG as a whole may be considered in addition to the presence ofabnormal QTc features for each respective patient.

Training may include using 5-fold cross-validation on a plurality ofmodels such as two machine learning models using the baseline ECGs ofapproximately 92,848 resulting pairs to predict drug-induced LQT (>500ms) in the on-drug ECGs. Artificial intelligence engines may beimplemented, including, by example, a deep neural network using ECGvoltage data and a gradient-boosted tree using the baseline QTc with ageand sex as additional inputs to both models. Other models may includeone or more inputs as described herein. Other combinations of folds,hold-out patients, validations, and number of models for comparison maybe considered without departing from the methodology as describedherein.

In one such training on an available patient dataset having paired ECGdata for patients with both an off-drug ECG and an on-drug ECG, on-drugLQT prevalence was 16%. The ECG model demonstrated superior performancein predicting on-drug LQT (area under the receiver operatingcharacteristic curve (AUC)=0.756) compared to the QTc model (0.710). Ata potential operating point such as depicted in FIG. 23 , the ECG modelhad 89% sensitivity and 95% negative predictive value. Even in thesubset of patients with baseline QTc <470/480 ms (male/female; post-drugLQT prevalence=14%), the ECG model demonstrated good performance(AUC=0.736). An ECG-based machine learning model can stratify patientsby risk of developing drug induced LQT better than a model usingbaseline QTc alone. This model may have clinical value to identifyhigh-risk drug starts that would benefit from closer monitoring andothers who are at low risk of drug induced LQT.

Patients having been identified as high risk for drug-induced LQT maythen be reported to their respective physicians for additionalmonitoring, potential therapy and treatment modifications, or otherrisk-reduction steps as determined by the physician. In one example, thereporting may include additional risk-reduction steps based upon one ormore personal characteristics of the patient, the patient's medicalhistory, the patient's ECG, or publications identified as beingpertinent to the patient based upon available data. In anotherembodiment, the high-risk identification may be generated real-time fromthe ECG equipment itself based upon the ongoing ECG and the patientcharacteristics uploaded to the equipment either manually by diagnosticpersonnel or retrieved from the patient's EMR linked to the ECGequipment.

For systems, methods, and devices described herein, additional cardiacevents may be considered for modeling and/or predictions independentlyor together with Afib. In one example, a composite model architecturemay be considered which provides an architecture for each cardiac eventprediction that a composite model system may operate.

Composite Model

A system and method for generating and applying a composite model isdisclosed herein. In some embodiments, the composite model is anECG-based machine-learning composite model. In some embodiments, thecomposite model can predict a composite heart disease endpoint orcardiac event. In some embodiments, a composite model yields a higherpositive outcome metric, such as a positive predictive value (PPV), tofacilitate more practical recommendation of echocardiography to improveunder-diagnosis of heart disease. In some embodiments, the compositemodel comprises an electrocardiogram (ECG)-based machine learningapproach to predict multiple heart disease endpoints simultaneously.

A composite model may be used, for example, to identify high-riskpatients. The composite model may use data more ubiquitously availablethan transthoracic echocardiograms (TTEs), such as 12-leadelectrocardiograms (ECGs). ECGs are far more common, inexpensive, andperformed for a much broader range of indications, including onasymptomatic patients (for example in the preoperative setting). Thecomposite model may thus serve as a screening tool such that patientsidentified as high risk could be referred for diagnostic TTE.

In some embodiments, the composite model may be used to identifypatients at high risk for any one of numerous heart disease endpointswithin a single ECG platform, including moderate or severe valvulardisease (aortic stenosis [AS], aortic regurgitation [AR], mitralstenosis [MS], mitral regurgitation [MR], tricuspid regurgitation [TR]),reduced left ventricular ejection fraction [EF], and increasedinterventricular septal [IVS] thickness. The composite model maygenerate a composite prediction with higher yield/PPV that wouldfacilitate a more practical clinical recommendation for follow-updiagnostic echocardiography.

Clinically, a composite model can enable targeted TTE screening to helpdetect unrecognized and underdiagnosed diseases. A composite model mayhave both high sensitivity and precision. The composite model can helpguide the decision to obtain a TTE even for asymptomatic patients,shifting the balance to a scenario where TTE can be effective as ascreening tool downstream of an ECG, and helping clinicians diagnosepatients at the right time to prevent downstream adverse events,optimize the timing of interventions, and better implementevidence-based monitoring or management.

A machine-learning composite model using only ECG-based inputs canpredict multiple important cardiac endpoints within a single platformwith both good performance and high PPV, thereby representing apractical tool with which to better target TTE to detect undiagnoseddisease. As shown in Example 1, below, an exemplary composite model isdescribed and confirmatory results through retrospective real-worlddeployment scenarios are provided, to show the large impact that such amodel can have on patients when deployed across a health system. Theseapproaches to both clinical predictions and simulated deploymentrepresent practical solutions for existing limitations in theimplementation of machine learning in healthcare.

In some embodiments, the machine learning composite model may be trainedto predict composite echocardiography-confirmed disease within a certainperiod of time. For example, the composite model may be trained topredict composite disease within 1 year. In some embodiments, themachine learning composite model may be trained to predict 2, 3, 4, 5,6, 7, or more diseases. For example, an exemplary composite model may betrained to predict moderate or severe valvular disease. As anotherexample, a composite model may be trained to predict one or more ofaortic stenosis, aortic regurgitation, mitral stenosis, mitralregurgitation, tricuspid regurgitation, abnormally reduced ejectionfraction, and abnormal interventricular septal thickness.

A composite model may be employed as part of a system described, forinstance, in U.S. Patent Publication No. 2021/0076960, titled ECG BasedFuture Atrial Fibrillation Predictor Systems and Methods, the contentsof which are incorporated herein by reference in their entirety for allpurposes.

Example 1

In one example, an ECG-based cardiovascular disease detection system mayemploy a machine-learning platform comprising a composite model whichcan effectively predict clinically significant valvular disease, reducedleft ventricular EF, and increased septal thickness with excellentperformance (AUROC 91.4%) by using only ECG traces, age, and sex.Furthermore, the combination of these distinct endpoints into a singleplatform tied to a recommendation for a singular, practical clinicalresponse-follow-up echocardiography-resulted in an overall PPV of 52.2%for a clinically meaningful disease while maintaining high sensitivity(90%) and specificity (75.5%). This novel approach of combining multipleendpoints which align in the same recommended clinical action enablesthe model to leverage the increased prevalence and probability of anyone disease state occurring to improve our predictive performance forpotential clinical implementation.

Moreover, this approach may have potential clinical utility in aretrospective deployment scenario. In one example, a retrospectivedeployment scenario was trained on data pre-existing relative to a firstpoint in time (e.g., data prior to 2010 until some data endpoint) anddeployed on all patients without prior disease who obtained an ECG in2010, maintaining similarly high performance as compared to the maincross-validation results based only on passive observation and standardclinical care. With an active deployment of the present platform, evenhigher yields/PPV may be achieved once clinicians can pursue activeintervention in the form of follow-up TTE or more detailedhistory-taking and physical examination based on the model.

Using 2,141,366 ECGs linked to structured echocardiography andelectronic health record data from 461,466 adults, a machine learningcomposite model was trained to predict compositeechocardiography-confirmed disease within 1 year. Seven exemplarydiseases were included in the composite label: moderate or severevalvular disease (aortic stenosis or regurgitation, mitral stenosis (MS)or regurgitation, tricuspid regurgitation), reduced ejection fraction(EF)<50%, or interventricular septal thickness >15 mm. In otherexamples, the model may be trained to predict otherechocardiography-confirmed diseases, and other clinical thresholdsbesides 50% for abnormal reduced ejection fraction or 15 mm for abnormalinterventricular septal thickness may be used. Composite modelperformance was evaluated using both 5-fold cross-validation and asimulated retrospective deployment scenario. Various combinations ofinput variables (demographics, labs, structured ECG data, ECG traces)were also tested. The composite model with age, sex and ECG traces hadan AUROC of 91.4% and a PPV of 52.2% at 90% sensitivity. Individualdisease model PPVs were lower, ranging from 2.1% for MS to 41.3% forreduced EF. A simulated retrospective deployment model had an AUC of88.8% on data trained pre-2010 and, when deployed on at-risk patients in2010, identified 22% of patients as high-risk with a PPV of 40%. TheAUROC for different variable inputs ranged from 84.7% to 93.2%.

Data was retrieved and processed from three clinical sources at a largeregional US health system (a first entity), including 2,091,158 patientsfrom the health system's electronic health record (EHR) (a firstsource), 568,802 TTEs from a second source, and 3,487,304 ECG tracesfrom a third source. In another embodiment, it will be understood thatdata may be obtained from a plurality of sources related to a pluralityof different or unrelated entities. From this data, all ECGs after afirst point in time (e.g., 1984) from patients 18 years old, sampled ateither 250 hz or 500 hz with at least 8 leads, and with a correspondingmedical record from the first source were included. This intersection ofthe first and third sources yielded 2,884,264 ECGs from 623,354patients.

Vitals, labs, and demographics as of the ECG acquisition time were alsoobtained. Table 4 lists inputs grouped by category, although it will beappreciated that the model may utilize one or more other inputs withinthe categories listed or within one or more other categories. Each inputis shown with its units in parenthesis. The ECG findings were binary.

TABLE 4 List of inputs Demographics Age (years), race (white/other),sex, smoke (ever), BMI (kg/m2), and Vitals diastolic and systolic bloodpressure (mmHg), heart rate (bpm), height (cm), weight (kg). Labs A1C(%), Bilirubin (mg/dl), BUN (mg/dl), Cholesterol (mg/dl), CKMB (ng/ml),Creatinine (mg/dl), CRP (mg/l), D dimer (mcg/ml FEU), Glucose (mg/dl),HDL (mg/dl), Hemoglobin (g/dl), LDH (u/l), LDL (mg/dl), Lymphocytes (%),Potassium (mmol/l), PRO BNP (pg/ml), Sodium (mmol/l), Troponin I and T(ng/ml), Triglyceride (mg/dl), Uric acid (mg/dl), VLDL (mg/dl), eGFR(ml/min/1.73 m²) ECG findings Acute MI, Afib, Aflutter, Complete Block,Early rep, Fas block, First deg block, Intrav Block, In Lbbb, In rbbb,Ischemia, Lad, Lbbb, Low QRS, LVH, Non-spec ST, Non-spec T, Normal,Other Brady, PAC, Pacemaker, Poor tracing, Prior infarct, Prior MIanterior, Prolonged QT, PVC, RAD, RBBB, Sec deg block, Sinus Brady, SVT,Tachy, T Inversion, Vtach ECG Avg RR interval (ms), PR interval (ms), Paxis, QRS duration (ms), QT measurements (ms), QTC (ms), R axis, T axis,Ventricular rate (bpm)

The closest past measurement to the ECG was used unless the measurementwas older than a year, in which case a missing value was assigned. TTEmeasurements and diagnoses (AS, AR, MR, MS, and TR) were extracted fromreports from the second source; and ECG structured findings,measurements, and 12-lead traces were extracted from the third source.ECGs were then labeled as detailed in the following sections, and ECGswithout a label were discarded for all disease outcomes. Overall,2,141,366 ECGs with at least 1 label from 461,466 patients were included(FIG. 24 ).

Specifically, FIG. 24 displays a block diagram of source data to datasetused for experiments described in this patent. First source (EHR) datawas processed into a cardiovascular pipeline to retrieve patients withphysical encounters in the first entity health care system or that haverecords of an ECG or Echocardiography study. The clinical database ofdata from the third source was processed into a database, such as alightning memory-mapped (LMDB) database, of ECGs sampled at either 250hz or 500 hz, having at least 8 leads, having an acquisition date stamplater than 1984, coming from patients older than 18 years (as reportedin the ECG), and with a cross-referenced medical record number (checkedagainst an EHR processed list from the first source). The no-label ECGsrefer to ECGs that did not meet any labeling criteria (AS, AR, MS, MR,TR, EF<50%, nor IVS>15 mm).

Labeling

TTE-Confirmed Disease Outcome Definitions

A plurality of outcome labels (e.g., 7 outcome labels) using TTEreports, one for each disease outcome of interest (AS, AR, MR, MS, TR,reduced EF, increased IVS thickness). String matching was used on thereports to identify the presence of valvular stenosis or regurgitation,as well as the associated severity level (Table 5). Specifically, Table5 includes a keyword list for assigning an abnormality and severity toeach valve in an Echocardiography report.

TABLE 5 Abnormality Stenosis stenosis, stenotic Regurgitationregurgitation, regurgitant, insufficiency Severity Normal absent, nostenosis, no AS, no MS, not stenotic, no PS, no tricuspid stenosis, nosignificant, no regurgitation, No TR, No MR, TS excluded, MS excluded,AS excluded, w/o stenosis, no mitral, no AR, trace, no evidence of, nopulmonic, no mitral, without aortic stenosis, stenosis is absent, nomitral regurgitation, physiologic, no hemodynamically, Normal 2-D,Normal MV, not sign, Normal structure and function, normal prosthetic,normal function, function normal, There is a normal amount of, isprobably normal, is normal without Mild mild, valvular, aortic stenosisis present, valve stenosis is present, stenosis is possible, stenosis ispossibly present, borderline Moderate moderate, Mod Severe severe,possibly, severe, moderate-severe, mod-severe, moderate - severe,moderately severe, moderate to severe, critical, consistent withsignificant Valve Aortic aortic, AS, AR, AV Tricuspid tricuspid, TR, TS,TV Mitral mitral, MR, MS, MV

Each of 5 valvular conditions of interest were labeled as positive ifmoderate or severe and negative if reported normal or mild in severity,or a missing label was otherwise assigned.

Reduced EF was defined as a TTE-reported EF of <50%, and increased IVSthickness as >15 mm, although it will be appreciated that other rangesfor EF and/or IVS thickness may be used to define reduced EF. TTEs notmeeting those criteria were labeled as negative, and a missing label wasassigned when the measurement was missing.

Outcome labels extracted from TTE reports for AS, AR, MR, MS, and TRwere manually validated using chart review of 100-200 random sampleswhere PPVs and NPVs of 98-100% were found.

ECG Labeling

An ECG was labeled as positive for a given outcome if it was acquired upto a first time period, e.g., one year, before or any time after (up toa censoring event) the patient's first positive TTE report. An ECG waslabeled as negative if it was acquired more than the first time period,e.g., one year prior to the last negative TTE or a censoring eventwithout any prior positive TTEs (FIG. 25A). Specifically, FIG. 25Adisplays the patient timeline used to label (I) positive ECGs (+ECG inplot I), (II) confirmed negative ECGs (−ECG in plot II), and (III)unconfirmed negative ECGs (−ECG in plot III). The censoring event inplots I and II in FIG. 25A are any intervention that could modify theunderlying physiology of the disease of interest. The last negative Echoensures no record of prior positive Echo exists. The bottom timeline isused for patients that never got an Echo. The censoring event in plotIII in FIG. 25A is defined as the last known patient encounter wherephysical presence is required.

Also, in the absence of any history of TTE, an ECG was also classifiedas negative if there was at least 1 year of subsequent follow-up withouta censoring event and no coded diagnoses for the relevant disease (Table6). Specifically, Table 6 lists ICD 10 codes used to search for evidenceof diagnosis in ECGs from patients that never had an Echo. A negativelabel was assigned if none of the codes were ever present in thepatient's chart.

TABLE 6 Diagnosis ICD10 codes AS I06.0, I06.2, I06.8, I06.9, I08.0,I08.2, I08.3, I08.8, I08.9, I35.0, I35.2, I35.8, I35.9, Z95.4, I33.*,Q20.*, Q21.*, Q22.*, Q23.*, Q24.* AR I06.1, I06.2, I06.8, I06.9, I08.0,I08.2, I08.3, I08.8, I08.9, I35.1, I35.2, I35.8, I35.9, Z95.4, I33.*,Q20.*, Q21.*, Q22.*, Q23.*, Q24.* MR I05.1, I05.2, I05.8, I05.9, I08.0,I08.1, I08.3, I08.8, I34.0, I34.1, I34.8, I34.9, Z95.4, I33.*, Q20.*,Q21.*, Q22.*, Q23.*, Q24.* MS I05.0, I05.2, I05.8, I05.9, I08.0, I08.1,I08.3, I08.8, I34.2, I34.8, I34.9, Z95.4, I33.*, Q20.*, Q21.*, Q22.*,Q23.*, Q24.* TR I07.1, I07.2, I07.8, I07.9, I08.1, I08.2, I08.3, I08.8,I36.1, I36.2, I36.8, I36.9, Z95.4, I33.*, Q20.*, Q21.*, Q22.*, Q23.*,Q24.* EF < 50% I42.0, I42.6, I42.7, I42.8, I42.9, T86.2, T86.3, Z94.1,Z94.3, I09.81, I97.13, I25.5, B33.2, O90.3, I43.*, I50.*, I51.8* IVS >15 mm I37.1, I37.2, Z95.4, E83.11, I10.*, I11.*, I12.*, I13.*, I15.*,I16.*, E85.*, I42.1*, I42.2*, Q20.*, Q21.*, Q22.*, Q23.*, Q24.*, Q25.*,E74.*, E75.*, D86.*

A censoring event was defined as death, end of observation, or anintervention that directly treated the disease and could modify theunderlying physiology or impact the ECG signal, such as valvereplacement or repair. In other embodiments, heart transplant or LVADstatus, for example, may be included as censoring events. A negative TTEreport after a positive TTE report also may be used as a censoring eventto account for the possibility of such interventions being performedoutside of the first entity healthcare system.

For the composite endpoint, an ECG was labeled as positive if any of theseven individual outcomes were positive and as negative if all sevenoutcomes were negative.

Model Development

A plurality of models, e.g., 8 models, may be developed using differentcombinations of multiple input sets including structured data(demographics, vitals, labs, structured ECG findings and measurements)and ECG voltage traces.

In one instance, for the ECG trace models, a low-parameter convolutionalneural network (CNN) was developed with 18,495 trainable parameters thatconsisted of six 1D CNN-Batch Normalization-ReLU (CBR) layer blocksfollowed by a two-layer multilayer perceptron and a final logisticoutput layer (Table 7). Specifically, Table 7 details a single outputlow-parameter CNN design for training on 8 non-derived ECG leads. Thenetwork contains a total of 18,945 trainable and 384 non-trainableparameters. Both Dropout layers were set at 25% drop rate. CBR is abrief notation for a sequence of 1D CNN, batch normalization, and ReLUlayers.

TABLE 7 Layer Output Shape #Parameters Input (5000, 8)  0 Rescaling(5000, 8)  0 CBR-1 (5000, 16)    656 + 64 CBR-2 (5000, 16)  1,296 + 64MaxPool1D (1666, 16)  0 CBR-3 (1666, 16)  1,296 + 64 CBR-4 (1666, 16) 1,296 + 64 MaxPool1D (555, 16) 0 CBR-5 (555, 16) 1,296 + 64 CBR-6 (555,16) 1,296 + 64 MaxPool1D (185, 16) 0 CBR-7 (185, 16) 1,296 + 64 CBR-8(185, 16) 1,296 + 64 MaxPool1D  (61, 16) 0 CBR-9  (61, 16) 1,296 + 64CBR-10  (61, 16) 1,296 + 64 MaxPool1D  (20, 16) 0 CBR-11  (20, 16)1,296 + 64 CBR-12  (20, 16) 1,296 + 64 MaxPool1D  (6, 16) 0 Flatten(96,)  0 Dense + Dropout (32,)  3104   Dense + Dropout (16,)  528  Dense(1,)  17 

Each CNN layer consisted of 16 kernels of size 5. The same networkconfiguration was used to train one model per clinical outcome,resulting in 7 independently trained CNN models (FIG. 25B).Specifically, FIG. 25B displays a block diagram for a composite modelthat shows the classification pipeline for ECG trace and other EHR data.The output of each neural network (the triangles in FIG. 25B) applied toECG trace data is concatenated to labs, vitals, and demographics to forma feature vector. The vector is the input to a classification pipeline(min-max scaling, mean imputation, and XGBoost classifier), whichoutputs a recommendation score for the patient.

To form the final composite model and combine ECG trace-based modelswith structured data, the risk scores resulting from the individual CNNswere concatenated with the structured data. The concatenated featurevector was used to train a classification pipeline consisting of amin-max scaler (min 0, max 1), mean imputation, and a machine learningmodel or gradient boosting library classifier such as an XGBoostclassifier, as shown in FIG. 25B.

Model Evaluation. The models were evaluated using two approaches, 1) atraditional random cross-validation partition, and 2) a retrospectivedeployment scenario where, using 2010 as the simulated deployment year,past data was used to train and future data was used to test. Area underreceiver operating characteristic curve (AUROC), area under theprecision-recall curve (AUPRC), and other performance metrics(sensitivity, specificity, positive and negative predictive values) weremeasured at multiple operating points (Youden, F1, F2, at 90% and 50%sensitivity, at 25% and 33% PPV).

Cross validation. A 5-fold cross-validation was followed by randomlysampling 5 mutually exclusive sets of patients. Each set was expanded toall ECGs from each patient to form the training and test ECG sets. Whentraining the CNN models for each individual endpoint, samples withmissing labels were discarded. The model was then applied to all testsamples—regardless of missingness of the true label—and marginalperformance was evaluated only on samples with complete labels that alsosatisfied the composite model labeling criteria described above.Performance statistics were reported as the average across the fivefolds (with a 95% confidence interval) in a random ECG per patient.

Retrospective deployment. In addition to the cross-validation approach,a deployment of the model was also retrospectively simulated using acutoff of the year 2010, re-labeling all ECGs with information availableas of Jan. 1, 2010. This artificially constrained dataset was used toreplicate the cross-validation experiments and train a deployment modelusing data prior to 2010. The deployment model then was applied to thefirst ECG per patient for all patients seen through Dec. 31, 2010.Performance statistics on all ECGs from patients at risk were measured,and the true outcomes of the at-risk population using all informationavailable as of May 4, 2021, were determined.

Results from Example 1

568,802 TTE reports were identified from 277,358 patients, of which150,730 were positive for at least one disease outcome label. Diseaseprevalence ranged from 0.7% for MS to 19.9% for reduced EF (Table 8).Specifically, Table 8 lists TTE label count and relative prevalence foreach diagnosis among the 568,802 TTEs.

TABLE 8 Normal-Mild Moderate-Severe Prevalence AS 271,384 21,790 7.4% AR278,439 13,878 4.7% MR 270,266 32,002 10.6% MS 302,649 2,188 0.7% TR258,236 36,069 12.3% False True EF < 50 308,695 76,806 19.9% IVS > 15362,974 27,389 7.0%

2,141,366 ECGs were identified from 461,466 patients who met criteriafor a positive or negative individual disease label (AS, AR, MS, MR, TR,EF, or IVS), of which 1,378,832 ECGs from 333,128 patients qualified forthe composite label (Table 9). Specifically, Table 9 lists the count ofECGs and total prevalence for each diagnosis among the 2,141,366 ECGswith at least a complete label. Confirmed counts are based on ECGs frompatients that also underwent an Echocardiography study that confirmedthe diagnosis. Unconfirmed negatives (-) show the count of ECGs frompatients that never got an Echocardiography and had no history of thedisease using the ICD code filters from Table 6.

TABLE 9 Negative Positive Prevalence No label AS 1,608,160 65,037 3.9%468,169 AR 1,609,710 58,209 3.5% 473,447 MR 1,536,378 145,355 8.6%459,633 MS 1,691,737 9,920 0.6% 439,709 TR 1,556,020 148,916 8.7%436,430 EF < 50% 1,375,507 315,874 18.7% 449,985 IVS > 15 mm 1,235,255121,583 9.0% 784,528 Present Composite 805,353 573,479 41.6% 762,534Model

Table 10 displays a breakdown by ECG label of each model feature.Specifically, Table 10 displays average value for each predictor groupedby whether they qualified for the composite labeled ECGs. False refersto ECGs from patients that were not diagnosed with any of the 7 diseaseswithin a year, and True to ECGs from patients that were diagnosed withat least one of the 7 diseases within a year or before the ECGacquisition time.

TABLE 10 FALSE TRUE Demographics and Vitals: Age 55.9 (16.9) 71.2 (13.6)Race 96.9 97.5 Sex 44.7 58.3 Smoker 58.3 62.9 BMI 30.8 (8.4) 30.1 (9.1)BP Diastolic 73.8 (11.2) 70.2 (12.8) BP Systolic 127.5 (18.6) 128.1(21.5) Heart Rate 77.1 (14.8) 76.1 (16.5) Height 168.3 (10.5) 168.9(11.2) Weight 87.2 (23.7) 86.0 (24.5) Labs: A1C 6.8 (3.9) 6.9 (1.6) BILI0.5 (0.5) 0.6 (0.7) BUN 16.5 (8.7) 25.6 (16.2) Cholesterol 183.6 (46.0)158.8 (47.4) CKMB 6.6 (23.5) 10.0 (37.2) Creatinine 1.0 (2.0) 1.4 (1.3)CRP 22.7 (51.0) 48.3 (70.6) Ddimer 1.0 (2.0) 1.9 (3.1) Glucose 114.0(43.3) 123.1 (52.1) HDL 49.6 (16.1) 45.6 (15.6) Hemoglobin 13.7 (21.9)13.9 (43.9) LDH 217.4 (134.6) 274.2 (290.8) LDL 103.4 (37.6) 85.7 (37.1)Lymphocytes 24.9 (10.5) 19.9 (10.6) Potassium 4.2 (0.7) 4.3 (0.7) PROBNP1032.1 (3880.0) 6868.1 (12317.4) Sodium 139.3 (3.0) 138.9 (3.7)TroponinI 0.8 (12.4) 1.1 (10.2) TroponinT 0.1 (0.4) 0.2 (1.1)Triglyceride 158.8 (127.7) 144.9 (108.9) UricAcid 6.1 (2.1) 7.1 (2.7)VLDL 30.4 (15.8) 28.1 (15.9) eGFR 58.0 (16.9) 49.8 (15.2) ECG: Avg RRInterval 831.5 (186.0) 794.2 (204.6) PR Interval 158.3 (32.6) 175.5(388.3) P Axis 47.7 (25.5) 50.3 (36.4) QRS Duration 90.6 (17.7) 109.6(30.8) QT 392.3 (43.9) 409.8 (59.5) QTC 433.5 (34.1) 463.6 (45.9) R Axis28.1 (40.7) 17.9 (64.3) T Axis 42.6 (37.4) 69.8 (69.8) Vent Rate 76.1(18.8) 80.8 (22.6) Acute MI 0.6 2 AFIB 3.1 18.4 Normal 52.6 28.4AFLUTTER 0.6 2.8 FAS Block 1.9 4.9 First Deg Block 3.8 9.3 Intrav Block0.8 5.2 In RBBB 0.1 0.9 Ischemia 5.4 18.3 LAD 5.7 14.8 LBBB 0.8 6.1LOWQRS 3.5 6.2 LVH 5.7 10.6 Non-Spec ST 8.2 13.4 Non-Spec T 13.5 19.9PVC 3.5 12.8 PAC 3.3 7 Pacemaker 1.3 10 Poor Tracing 4.1 6.5 PriorInfarct 12.5 28.6 Prior MI Ant. 4.6 12.2 Prolonged QT 3.2 8.4 RAD 1.9 3RBBB 3.3 9.8 Sinus Brady 15.6 10.8 Tachy 7.9 7.9 T Inversion 2.7 7.5

At baseline, across 2.14 million ECGs, the median patient age was 64.7,50.4% were male, and 96.7% were white (Table 11). Specifically, Table 11lists features extracted at the time of the ECG and their overallaverage, for continuous values, or prevalence, for binary values. OtherECG features not listed because of their rarity (<1%) were: CompleteBlock, Other Brady, Early Rep, IN LBBB, Sec Deg Block, SVT, and VTACH.ECG findings showed 43.5% were normal, 8.3% had atrial fibrillation,1.0% showed acute myocardial infarction, and 7.7% showed leftventricular hypertrophy.

TABLE 11 Mean Median [IQR] Demographics and Vitals: Age (years) 63 64.7[52, 76] Race (% White) 96.7% Sex (% Male) 50.4% Smoke (% Ever) 59.1%BMI (kg/m2) 30.7 29.4 [25, 35] Dias. BP (mmHg) 72.6 72 [64, 80] Sys. BP(mmHg) 128.8 128 [116, 140] Heart Rate (bpm) 76.9 75 [66, 85] Height(cm) 168.6 167.6 [160, 178] Weight (kg) 87.2 84.1 [70, 100] Labs: A1C6.8 6.4 [5.7, 7.5] BILI 0.6 0.5 [0.3, 0.7] BUN 20.3 17 [13, 23]Cholesterol 171.6 167 [139, 199] CKMB 8.3 2.9 [1.8, 4.9] Creatinine 1.20.9 [0.8, 1.2] CRP 36.5 9 [2.5, 39.0] D dimer 1.5 0.6 [0.3, 1.5] Glucose118.4 103 [93, 125] HDL 47.9 45 [37, 56] Hemoglobin 13.6 13.1 [11.6,14.3] LDH 258 211 [173, 272] LDL 94.6 90 [68, 117] Lymphocytes 22.4 22[14.8, 29] Potassium 4.2 4.2 [3.9, 4.5] PROBNP 4351 1015 [249, 3553]Sodium 139.2 140 [137, 141] Troponin I 88.8 3 [1.2, 5] Troponin T 13.5 1[1, 3.8] Triglyceride 152.2 125 [89, 181] Uric Acid 6.6 6.2 [4.9, 7.9]VLDL 28.6 25 [18, 35] eGFR 54.5 60 [55.2, 60] ECG: Avg RR Interval 813.5806 [678, 938] PR Interval 164.4 160 [142, 180] P Axis 48.5 51 [34, 65]QRS Duration 97 90 [82, 102] QT 397.9 396 [366, 428] QTC 444.6 440 [419,464] R Axis 22.9 21 [−9, 54] T Axis 51.6 45 [23, 70] Vent Rate 78.2 74[64, 88] Acute MI 1.0% AFIB 8.3% Normal 43.5% AFLUTTER 1.3% FAS Block3.2% First Deg Block 6.1% Intrav Block 2.1% INRBBB 3.3% Ischemia 9.6%LAD 9.1% LBBB 2.6% LOWQRS 4.7% LVH 7.7% Non-Spec ST 10.4% Non-Spec T16.0% PVC 6.7% PAC 5.1% Pacemaker 4.2% Poor Tracing 5.3% Prior Infarct18.4% Prior MI Ant. 7.3% Prolonged QT 5.0% RAD 2.2% RBBB 6.0% SinusBrady 13.5% Tachy 8.4% T Inversion 4.5%

Composite Model Input Evaluation

Table 12 shows the results of 5-fold cross validation comparingcomposite model performance as a function of different input features.Specifically, Table 12 provides a performance comparison ofcross-validated models with varying input features for the compositeendpoint (valve disease, reduced EF, increased IVS). All values areshown in percentage with the 95% CI in between brackets. Each model wastested on a random ECG per patient. The AUROC ranged from 84.7 [95% CI.84.5,85.0] for the model built only with structured ECG findings andmeasurements to 93.2 [93.0,93.4] for the model with all available inputs(structured ECG findings and measurements, demographics, labs, vitals,and ECG traces). While the model with all available inputs provided thebest performance, the remainder of the results focus on models thatinclude only age, sex, and ECG traces since this input set is readilyavailable from the third entity or other ECG systems and best balancesportability and performance.

TABLE 12 Input ROC-AUC PRC-AUC PPV@90% Sens. Spec.@90% Sens. A) ECGFindings and Meas. 84.7 [84.5, 85.0] 67.5 [67.0, 67.9] 36.1 [35.7, 36.5]52.8 [52.0, 53.6] B) Demo., Labs, and Vitals 87.9 [87.7, 88.1] 72.9[72.4, 73.4] 43.0 [42.8, 43.1] 64.4 [64.2, 64.6] C) ECG Traces 91.0[90.7, 91.4] 77.6 [76.8, 78.5] 50.8 [49.8, 51.7] 74.0 [73.0, 74.9] A + C91.3 [91.0, 91.5] 78.3 [77.5, 79.1] 51.5 [50.7, 52.3] 74.7 [74.0, 75.5]Age + Sex + C 91.4 [91.1, 91.7] 77.5 [76.6, 78.5] 52.2 [51.3, 53.0] 75.5[74.7, 76.2] A + B 91.5 [91.4, 91.7] 79.7 [79.1, 80.2] 51.6 [51.0, 52.1]74.8 [74.3, 75.3] B + C 93.1 [92.8, 93.3] 82.7 [82.1, 83.3] 57.0 [56.1,58.0] 79.8 [79.1, 80.5] A + B + C 93.2 [93.0, 93.4] 83.0 [82.5, 83.5]57.5 [56.3, 58.6] 80.1 [79.3, 81.0]

Cross-Validation Performance of Composite Model

The composite model with age, sex, and ECG traces as inputs yielded anAUROC of 91.4 [91.1, 91.7] and a PPV of 52.2% [51.3, 53.0] at 90%sensitivity (Table 3*). Specifically, Table 13 displays ECG traces onlymodel results for cross-validation experiments. Results are shown at arandom ECG per patient and averaged across 5 folds. All values are shownin percentage with the 95% CI in between brackets. The any label ispositive when any of the other seven is positive, and negative when allthe other seven are negative.

TABLE 13 Prevalence ROC-AUC PRC-AUC PPV@90% Sens. Spec.@90% Sens. AS 3.7[3.6, 3.8] 92.4 [92.0, 92.8] 35.0 [32.4, 37.8] 14.7 [13.9, 15.6] 80.1[78.8, 81.3] AR 2.9 [2.8, 2.9] 87.5 [87.0, 88.0] 21.1 [19.1, 23.2] 7.2[6.9, 7.5]  65.9 [63.8, 67.9] MR 6.9 [6.8, 7.0] 92.2 [91.8, 92.6] 51.9[50.0, 53.8] 24.2 [22.7, 25.6] 79.0 [77.4, 80.5] MS 0.4 [0.4, 0.5] 92.3[90.5, 93.8] 7.2 [4.9, 10.5] 2.1 [1.4, 3.0]  80.7 [72.5, 86.9] IR 7.3[7.2, 7.3] 92.6 [92.0, 93.1] 57.2 [55.6, 58.7] 26.0 [24.3, 27.7] 79.9[78.1, 81.6] EF < 50%  13.0 [12.8, 13.1] 93.0 [92.3, 93.6] 70.5 [66.1,74.5] 41.3 [38.4, 44.2] 80.9 [78.6, 83.0] IVS > 15 mm 6.2 [6.1, 6.3]89.1 [88.9, 89.3] 36.7 [35.6, 37.8] 17.1 [16.6, 17.6] 71.2 [70.4, 72.0]Present Composite  22.9 [22.8, 23.1] 91.4 [91.1, 91.7] 77.5 [76.6, 78.5]52.2 [51.3, 53.0] 75.5 [74.7, 76.2] Model With Age, Sex, and ECG Tracesas inputs

The composite model yielded a significantly higher PPV than any of the 7models trained for an individual component endpoint, with the individualmodel PPVs ranging from 2.1% [1.4, 3.0] for MS to 41.3% [38.4, 44.2] forreduced EF (Table 13). The same trend was found for the AUPRC of thecomposite model, which was 77.5% [76.6, 78.5], compared to theindividual models ranging from 7.2% [4.9, 10.5] for MS to 70.5% [66.1,74.5] for EF (FIG. 26 ). Specifically, FIG. 26 displays an area underthe Precision-Recall curve for each of the individual diseases and themodel of the present disclosure. The dashed line shows the prevalencefor each of the labels.

Performance metrics for alternate composite model operating points arepresented in Table 14. Specifically, Table 14 lists composite modelperformance metrics at multiple threshold values.

TABLE 14 Threshold NPV PPV Sensitivity Specificity Value 0.1 95.8 [95.7,95.9] 54.4 [53.7, 55.2] 88.5 [88.1, 88.8] 78.0 [77.4, 78.5] 0.1 0.2 92.8[92.6, 93.0] 68.1 [67.4, 68.8] 76.9 [76.1, 77.6] 89.3 [89.0, 89.6] 0.20.3 90.3 [90.0, 90.5] 75.9 [75.2, 76.6] 66.0 [64.9, 67.1] 93.8 [93.6,93.9] 0.3 0.4 87.6 [87.4, 87.8] 81.4 [80.6, 82.3] 54.1 [53.3, 54.8] 96.3[96.1, 96.5] 0.4 0.5 84.8 [84.7, 85.0] 85.9 [84.9, 86.8] 41.1 [40.6,41.5] 98.0 [97.8, 98.1] 0.5 0.6 82.3 [82.0, 82.5] 89.3 [88.5, 90.1] 28.2[26.6, 29.9] 99.0 [98.9, 99.1] 0.6 0.7 79.8 [79.3, 80.3] 91.6 [91.0,92.2] 15.0 [11.6, 19.3] 99.6 [99.5, 99.7] 0.7 0.8 77.8 [77.5, 78.0] 94.2[92.6, 95.5] 3.5 [1.6, 7.4]   99.9 [99.9, 100.0] 0.8 0.9 77.1 [76.9,77.2]  0.0 [0.0, 100.0] 0.0 [0.0, 0.0]   100.0 [100.0, 100.0] 0.9 Youden94.4 [94.2, 94.6] 61.6 [60.2, 63.0] 83.2 [82.6, 83.9] 84.6 [83.6, 85.5]14.4 [13.5, 15.3] F1 92.5 [92.1, 92.9] 69.4 [67.2, 71.6] 75.5 [73.8,77.2] 90.1 [88.8, 91.2] 21.4 [19.6, 23.3] F2 95.9 [95.7, 96.0] 54.1[53.3, 54.9] 88.7 [88.2, 89.2] 77.6 [76.7, 78.4] 9.8 [9.2, 10.4] @25%PPV 99.5 [99.4, 99.6] 25.0 [25.0, 25.0] 99.8 [99.7, 99.9] 10.9 [10.2,11.5] 0.7 [0.6, 0.8] @33% PPV 98.8 [98.7, 98.9] 33.0 [33.0, 33.0] 98.3[98.2, 98.5] 40.6 [40.1, 41.0] 2.2 [2.1, 2.4] @90% Spec. 92.5 [92.3,92.8] 69.2 [68.8, 69.6] 75.6 [74.5, 76.6] 90.0 [90.0, 90.0] 21.2 [20.7,21.6] @50% Sens. 86.7 [86.6, 86.8] 83.0 [81.9, 84.1] 50.0 [50.0, 50.0]97.0 [96.7, 97.2] 43.3 [42.7, 43.8] @90% Sens. 96.2 [96.2, 96.2] 52.2[51.3, 53.0] 90.0 [90.0, 90.0] 75.5 [74.7, 76.2] 8.9 [8.6, 9.2]

Simulated Deployment Performance of Composite Model

As of Jan. 1, 2010, 563,375 ECGs were identified with a qualifying labelfor any of the seven clinical outcomes prior to 2010, of which 349,675ECGs qualified for the composite label to train the deployment model. A“qualifying” label was one that met the criteria for the applicableoutcome label. A cross-validation experiment within this data subsetshowed similar, yet slightly reduced performance of the composite modelcompared with the full dataset (AUROC 88.8 [88.5, 89.1]; PPV=44.0%[42.9, 45.1] at 90% sensitivity; Table 15). Specifically, Table 15 listscross-validation performance metrics computed with data prior to 2010.The five-fold average threshold that yielded 90% Sensitivity (0.056 froma range of 0 to 1) was taken to produce binary predictions on thedeployment model.

TABLE 15 Prevalence ROC-AUC PRC-AUC PPV@90% Sens. Spec.@90% Sens. AS 2.5[2.3, 2.6] 90.6 [89.7, 91.4] 22.8 [19.0, 27.1] 8.1 [7.4, 8.9] 74.1[71.5, 76.5] AR 2.8 [2.7, 2.9] 84.5 [83.5, 85.5] 15.6 [14.2, 17.1] 6.1[5.6, 6.6] 60.1 [55.6, 64.5] MR 7.0 [6.8, 7.3] 89.3 [88.2, 90.2] 40.1[36.0, 44.2] 19.6 [17.6, 21.7] 72.0 [68.6, 75.2] MS 0.3 [0.2, 0.3] 88.1[84.3, 91.1] 3.8 [1.8, 7.7]  0.7 [0.4, 1.2] 65.0 [46.6, 79.7] TR 5.4[5.2, 5.6] 90.6 [89.9, 91.2] 41.2 [38.3, 44.1] 16.7 [15.3, 18.1] 74.3[71.6, 76.8] EF < 50%  12.3 [12.0, 12.6] 90.7 [87.7, 93.0] 57.5 [46.7,67.7] 35.6 [30.2, 41.4] 77.1 [71.0, 82.3] IVS > 15 mm 7.2 [7.1, 7.4]85.9 [85.1, 86.7] 32.4 [31.4, 33.5] 16.3 [15.1, 17.6] 64.1 [60.6, 67.5]Composite  21.1 [20.9, 21.3] 88.8 [88.5, 89.1] 67.6 [66.3, 68.9] 44.0[42.9, 45.1] 69.4 [68.0, 70.8] Model

The deployment dataset contained ECGs from 69,465 patients (FIG. 27B).Of these, 5,730 patients were diagnosed with one of the seven clinicaloutcomes prior to 2010. This resulted in 63,735 at-risk patientsidentified between January 1^(st) and December 31^(st) of 2010. Usingthe previously determined threshold noted above, the deployment modellabeled 22.2% of patients as high risk for any of the seven diseaseoutcomes and 77.8% of patients as not high risk. Among the 4,642predicted high-risk patients with adequate follow-up who met our definedcriteria for the composite label, 1,867 patients truly developed one ofthe outcomes, yielding a PPV of 40.2%. Of these 1,867 patients, 231(12.4%) developed AS, 147 (7.9%) developed AR, 562 (30.1%) developed MR,32 (1.7%) developed MS, 505 (27%) developed TR, 1074 (57.5%) developedlow EF, and 460 (24.6%) developed IVS thickening—noting that 1083developed 1 of the 7 diseases while 496 developed 2, 225 developed 3, 55developed 4, 7 developed 5, 1 developed 6 and 0 developed all 7diseases.

Among those predicted not high risk, 27,648 patients did not develop anyof the outcomes within a year, for an NPV of 95.7%. At the patientlevel, for every 100 at-risk patients who obtained an ECG, the modelused with the present system and methods would identify 22 as high-risk,of which 9 would truly have disease, and 78 as not-high risk, of which75 would truly not have disease within 1 year (FIG. 27A). Specifically,FIG. 27A displays patient-level retrospective deployment results from2010 according to the present composite model. FIG. 27B displays aSankey plot of retrospective deployment results.

Outcome labels for 30,335 patients were undefined due to inadequatefollow-up or patients not meeting defined criteria for the compositelabel, as noted above. However, baseline characteristics among theseundefined patients and patients with complete outcome labels weresimilar (Table 16). Specifically, Table 16 displays baselinecharacteristics of patients with resolved vs unresolved labels indeployment scenarios. The AUROC among resolved labels was 84.4.

TABLE 16 Resolved Unresolved Mean(SD) Median Mean(SD) Median Age 56 (17)57 63 (17) 64 BMI 31 (8) 29 31 (8) 30 BP Diastolic 73 (11) 72 74 (12) 73BP Systolic 127 (18) 124 130 (19) 128 Heart Rate 76 (13) 74 75 (14) 74Height 168 (10) 168 168 (11) 168 Weight 87 (23) 84 87 (25) 84 A1C 7 (1)6 7 (1) 6 BILI 1 (1) 0 1 (1) 0 BUN 17 (9) 15 19 (11) 17 Cholesterol 182(43) 178 178 (43) 173 CKMB 4 (8) 3 5 (10) 3 Creatinine 1 (1) 1 1 (1) 1CRP 17 (40) 4 21 (45) 5 Ddimer 1 (3) 0 2 (3) 1 Glucose 109 (36) 99 111(36) 100 HDL 51 (16) 48 50 (15) 48 Hemoglobin 14 (28) 14 15 (50) 13 LDH217 (106) 191 246 (239) 197 LDL 102 (36) 98 98 (35) 94 Lymphocytes 25(10) 25 24 (11) 23 Potassium 4 (0) 4 4 (0) 4 PROBNP 2408 (8570) 418 1577(2868) 483 Sodium 139 (3) 139 139 (3) 139 TroponinI 0 (0) 0 0 (1) 0TroponinT 0 (0) 0 0 (0) 0 Triglyceride 150 (105) 126 151 (115) 127UricAcid 6 (2) 6 7 (3) 6 VLDL 30 (15) 27 24 (11) 21 eGFR 58 (8) 60 56(9) 60

The composite model described in Example 1 with results of 91.4% AUROC,52.2% PPV and 90% sensitivity on cross-validation is based on age, sex,and ECG traces alone as inputs, which may represent one possiblefavorable balance between performance and portability. This model usesdata readily available from any ECG system, including those systemscommonly available to and/or recognized by those of ordinary skill inthe art, so that it can easily be deployed across most healthcaresystems. Although the model substantially outperformed those using onlydemographics or structured ECG findings and measurements, it will beappreciated that other demographics/vitals, labs, ECG findings, and/orECG measurements, including any of the options listed in Table 1 orother relevant options may be used as inputs to train and/or deploy thecomposite model. While the addition of EHR data did slightly improveperformance, the inclusion of EHR data in some instances may result indecreased portability with the need for EHR or clinical data warehouseintegration. Thus, implementation of the present composite model mayrepresent a balance between marginal improvements in performance due tothe inclusion of different or additional inputs versus the time orprocessing costs associated with the integration, normalization,structuring, and/or other processing of additional or alternativeinputs.

In a simulated retrospective deployment on ECGs from 2010, approximately22% of at-risk patients without history of disease were predicted to behigh-risk for diagnosis of one of the seven cardiovascular diseaseoutcomes within the following year. Of the patients who were predictedhigh risk and had adequate follow-up, over 40% were truly diagnosed withdisease in the following year after index ECG, through only standardclinical care at the time and without any potential clinician behaviorchange or active intervention that true deployment of such a predictionmodel or decision support tool may elicit. This suggests that this 40%PPV is most likely a lower bound for the expected real-world performanceof the composite model described in Example 1. Meanwhile the 95.7% NPVsuggests that little disease will be missed but even in this case, themodel would not change what would otherwise be the clinical course forthese patients. Clinician behavior may change with a negative predictionif they are falsely reassured that the patient does not have disease orchanges their pretest probability and clinical reasoning. Thus,implementation can be designed so that clinicians are only alerted whena patient is predicted to be high risk, and for those patients, thereal-world data discussed herein indicates that more than 4 out of every10 patients will have true disease. Cross-validation performance metricsthat depend on prevalence (PPV, NPV, and AUPRC) may overestimatereal-world performance given the lower incidence or prevalence acrossthe generally smaller time window of deployment as opposed to thetypically extensive period used in cross-validation. For example, PPV incross-validation of the model disclosed herein was 52% but dropped to40% in simulated deployment. However, even a 40% increase in theidentification and potential for treatment of patients that ultimatelyexperience one or more of the modeled disease states still represents amarked-improvement over situations in which the disease states are notidentified until later on, e.g., once the patient has begun experiencingsymptoms.

The exemplary composite model described in Example 1 has somecharacteristics that need not be present in other embodiments. Forexample, the training and evaluation related to that composite modelwere limited to a single regional health system where most patients arewhite, so similar models designed and implemented according to thepresent disclosure may consider a diversity of the relevant patientpopulation and may factor that diversity into the relevant compositemodel or may adjust the present composite model to account for thatdiversity. Other models may consider and account for other differencesin patient populations, such as physiologic differences across raceand/or ethnicity to determine whether these ECG-based models performdifferently across groups. In addition, echocardiography-confirmeddiagnoses were used to generate the positive labels discussed herein,which were confirmed on chart review to have a high PPV. There may beadditional patients with disease-false negatives-who were not capturedusing this method, although the retrospective deployment discussedherein suggests that the negatives may be overwhelmingly true negativesas compared to false negatives, given the low prevalence of disease.Certain machine-learning approaches may have limited interpretability inidentifying feature importance. For example, IVS thickness may representinfiltrative diseases or may represent very poorly controlledhypertension. However, these diseases are important to recognize. Thus,model selection may take interpretability into consideration whenidentification is desired.

Composite Model Categorization and Implementation

Embodiments disclosed herein may also be presented as a backendrequiring minimal or no interaction from users of the system and may beentirely contained or compatible within an external electronic healthrecord system or electronic medical system. A third party system housingthe medical records of a hospital, physician office, institution,clinical trial, or other entity that manages patient data mayincorporate the embodiments as disclosed herein. The backend may beselected by administrators of the EMR/EHR and the underlying algorithmsautomatically applied on the integrated patient data.

A form may be provided within the EMR/EHR listing all availablecomposite models, such as algorithms predicting the risk of cardiacevents for a patient. The form may be presented as a website or as adynamically updated interface within the EMR/EHR. The form may includeone or more algorithm titles, an indication of a creating entity, adescription of the algorithm's functionality, optional indexable and/orfilterable tags, a description of the inputs to the algorithm, and adescription of the output the algorithm will provide. In someembodiments, the output may include a notification and/or the automaticinclusion of the output data into each respective patient's dataset.Each algorithm may be associated with a subscription fee set by thecreating entity. Exemplary composite models for generating one or moreform algorithms for listing are described herein.

A health care provider may enroll one or more databases of patients intothe system. Enrollment may include a backend, frontend, or other EMRintegration. The databases may exist within the confinement of the EMRsystem or may be uploaded to the cloud as part of an informationmanagement system. A secure data exchange system may interface and/orliaison information between the databases of patient information and oneor more databases of the system provider. Some healthcare providers maydesire to keep their databases separate from the system for patientprivacy and to protect their proprietary collections of data. Data maybe ingested in an unstructured manner for abstraction and curation, in astructured manner, and/or may be normalized between one or more data orstructure types. Data may also be securely exchanged between thestructured and/or normalized data and one or more of the databases ofpatient information and one or more databases of the system provider.

Example 2

In another embodiment, a stand-alone model or a composite model maypredict active aortic stenosis (AS) and/or AS within a time period suchas one year with accuracy using only ECG traces and patient featureswhich may be extracted from the patient's electronic health records.Other time periods between a few minutes, days, weeks, months, or yearsmay be implemented using the same modeling architectures and trainingmethods disclosed herein. Patient features may include age, gender,weight, height, blood pressure, diagnostic laboratory results, and otherfeatures. Composite Model training data generation may be modulatedbetween differing normalizing schemes for improved accuracy. Forexample, lead sample sizes or rates may be lengthened or shortened,patient features may be relabeled from their original states to one ormore of categorical states such as, in the case of age, 1-10, 11-20,21-30, and so forth. Other features may be categorized as well.

The composite model may be any machine learning algorithm which analyzesan ECG waveform and basic demographic data (age/sex) readily availablein a digital ECG file and consist of a deep neural network (DNN) orconvolutional NN (CNN) trained on data from hundreds of thousands ofpatients and echocardiograms, and more than a million ECGs. Thecomposite model may effectively analyze the time-voltage signals from adigital 12-lead ECG, with the addition of age and sex as input featuresinto the network, to yield a predicted high-risk score (probabilityestimate) for moderate or severe TTE-confirmable AS within 1 year of theECG.

Training a CNN to predict patients of high risk for AS within one yearof an ECG is more challenging than training a CNN to recognize activeAS. For example, the presentation of active AS, such as a patient whowill be diagnosed with AS should a physician review their ECG orechocardiography presents with significantly distinguishable ECGpatterns. However, recognizing patients with an ECG up to one year outinvolves identifying a characteristic from the ECG may be difficult todistinguish from the overbearing signals of active AS for an artificialintelligence engine such as a CNN. Compensating for the disparitybetween characteristics indicative of active AS and future AS may beimplemented during the training of the CNN. In one example, identifyingpatients having active AS from the training dataset and patients havingfuture AS from the dataset and assigning a label which is not suppliedas a training input may be implemented. Active AS may be expanded fromwithin a few days or weeks of an echocardiogram-based diagnosis whilefuture, or incidence, AS may include ECGs outside of those with activeAS and up to a year before diagnosis with a similar period expansion ofa few days or weeks. Training the CNN may include providing all patientstogether, without a distinguishing identifier between those havingactive AS and those with incidence or future AS. The internal validationset, hold out set, and/or test set are filtered to include only thosepatients which have the incidence or future AS label. By training themodel to recognize both active and future AS while refining the model torecognize future and incidence AS, the model may compensate for thedisparity between the strength of the signals which identify active ASand future AS and improve performance.

To account for the potential of AS to develop over time, labeling mayalso include identification of patients who have never had an occurrenceof AS in their EMR, such as those indicated by ICD codes related to AS,and may be extended to patients having ECGs but no echocardiography witha diagnosis of AS during the identification of patients having anegative label to be used in training the CNN.

In one example, ECG data may be segmented into multiple portions. Amodel may be trained using all portions of the segmented ECG or with avarying number of portions. For example, the first or last portion ofthe ECG may be removed to avoid artifacts that may be present.

In another example, features supplied to a model may be ranked, such asby degree of variance within the dataset or degree of importance asdetermined after training. Model training may then be performed only onthe top 10, 20, 50, 200, or more features as determined by the rankingmethod applied.

Avoiding bias within the dataset may be implemented by training withoutconfounding features when validation shows that a bias exists. In onesuch example, a dataset may become biased on the type of machineperforming the ECG, individual hospitals, financial access to healthcaresystems, race, or other biases. Internal validations may show that nosuch bias exists, and the training may include all features within thedataset.

The ECG inputs may be arranged within a matrix of data points having anumber of samples of each lead stored in rows or columns. The additionalpatient features may be input into the model downstream of the inputlayers of the NN at, for example, an XGBoost model component and/orconcatenation layer before generating a result at an endpoint, forexample, a series of fully connected layers.

While the training methods, bias avoidance methods, and other modelimprovements are described with respect to the prediction of patientshaving AS within one year, these methods may be implemented for otherhigh risk of present or future cardiac events, including each of thecomposite models described herein.

A patient identified as high-risk may prompt a notification, either atthe time of diagnostic testing using the ECG or during a patientevaluation period where the patient's records are scanned for high-riskevents. Upon notification, a physician may consider increased, moreaggressive monitoring for their patient or request the patient receivemore diagnostic testing, such as imaging of their heart via anultrasound/echocardiography.

Example 3

Cardiac amyloidosis (CA) is a rare and potentially fatal disease wherediagnosis is often delayed or mishandled due to fragmented guidelinesand a poor understanding of the disease etiology. If detected andtreated appropriately, prognosis can be improved considerably.

Initial modeling efforts may be stifled due to the lack of data sourcesfor patients having each of ECG history and records, patient features,and an identified presence or absence of CA. Efforts to generate areliable composite model may include applying machine learningalgorithms.

Preparing patients for model training may include generating binarylabels were to indicate whether a patient has CA based upon clinicianexpertise, confounding disease etiologies and data limitations. Caseswhich may indicate patients with CA which includes light-chain (AL),wild-type transthyretin (wtATTR), hereditary transthyretin amyloidosis(hATTR), and other CA-related features. In another example, patientlabels may be based on one or more time periods and/or utilize thepatient's entire health record. One time period may include patienthealth record features which are present within five years of adiagnosis of CA whether before or after. Control patients, or those witha negative label, may be patients who have not had a CA diagnosis in afive year period. Labeling may include a temporal sensing element up tothe date of diagnosis, such as having an upper and/or lower bound fortime periods which are anchored from the point of diagnosis of apatient. The time periods may be days, weeks, months, years, and in somecases may be extended to the entirety of the patient's data, such as alldata, within any time period, after or before the diagnosis. Additionalcomorbidities which may improve model performance through exclusion fromthe training dataset include screening patients having cerebralangiopathy diagnosis, AL (continuum of blood cancer), end-stage renaldisease (can cause an amyloid looking heart), cerebral amyloidangiopathy, or advanced hypertension. These comorbidities (or distinctdiagnosis) may present too similarly to CA and reduce model performanceif included. In other models, training may be refined through the use ofattention-based models or other approaches to emphasize the distinctionbetween competing diagnoses.

In one example, a composite model may include an ensemble of machinelearning and/or deep learning models trained from heterogeneous dataincluding but not limited to ECG, demographics, labs and vitals. Due tothe low number of CA patients, this ensemble may utilize deep-learningbased ECG feature extractor(s), trained on a clinically informedoutcome(s) of interest. As an example, large interventricular septalthickness (IVSD) is a hallmark of CA and is routinely examined indisease diagnosis. IVSD is distilled from echocardiogram making it ahighly constrictive screening tool requirement. Thereby the compositemodel circumvents this requirement by predicting IVSD from ECG,maintaining a larger pool of prospective patients. The proposed featureextractor is not exclusive to IVSD but applicable to any outcome(s) ofinterest predictable from ECG. These predictions can be used asstandalone CA risk scores or nested with other patient features to yielda CA specific model (FIG. 28A-28B).

FIGS. 28A-28B illustrate two potential configurations of an architecturesupporting a composite model for predicting high-risk patients for CA.Other configurations include 2D convolution layers, multiple deep modelsas feature extractors and one-shot learning approaches.

Example 3 Results

On a holdout set of amyloid patients, the following results wereobserved using an ECG classifier built to predict interventricularseptal thickness (IVSD) > or <=15 mm. All reported metrics are averagedover 5 folds with a 20× bootstrap for patient level metrics.

Results:

ROC AUC: 0.92+/−0.03

Sensitivity: 0.94+/−0.05

Specificity: 0.73+/−0.06

PPV: 0.73+/−0.05

A physician, upon receiving a notification a patient is labeled ashigh-risk CA, may pursue more aggressive monitoring, increaseddiagnostics testing, or consider the patient for treatment.

In one embodiment, an entity such as a pharmaceutical company may applythe composite model within the clinic to identify patients at a highlikelihood of having CA for the purposes of filling out clinical trialsor drug trials. This benefits the patients by decreasing the time todiagnosis and is of low cost to physicians (echo and PYP scans should becovered by insurance).

Example 4

Stroke is relatively common for cardiac events at approximately 800,000incident strokes in the US annually and presents with a high morbidityand mortality. Upon occurrence, there are major implications forfunctional status and disability which lead to annual cost of ˜$50Billion (US). Lifetime risk across patients may be as high as 1 in every4.

Developing a model may include referencing ECG data, EMR data, andproviding them to a composite model as described herein. EMR data mayinclude, above and beyond features such as age and gender,identification of symptoms including: unilateral weakness or sensorydeficit, facial droop, visual field defect, difficultyspeaking/understanding, diplopia, dysarthria, dysphagia, vertigo,incoordination. EMR data may also include diagnosis or comorbiditiessuch as hemorrhage/hematoma, complex migraine, seizure, braininfection/abscess, tumor, hypoglycemia; or even diagnosis and billingcodes within the EMR. In a composite model similar to those describedabove, three inputs may be provided to the model such as ECG leads, age,and gender. In another embodiment, a plurality of features may beselected, such as age, stroke phenotyping, atrial fibrillation (AF),HTN, HLD, DM, smoking, structural heart disease, endocarditis, TIA, PAD,and/or physical inactivity. Stroke phenotyping may include three(qualitatively different) categories: Acute in-system visit (e.g. asreferenced in a clinical text/notes showing patient is in ER fortreatment of a stroke that just happened), Acute out-of-system visit(e.g. patient has follow-up to recent stroke, <30 days ago), andHistorical stroke (e.g. patient mentions a stroke they had years ago).

A composite model for stroke high-risk prediction may include providing:

Inputs:

Clinical notes text: Aggregated to the episode level, and “vectorized”into a bag-of-words representation

EHR fields: Length of text and number of notes in the episode, admissiontypes (e.g. ER or Urgent Care), basic “grayzone” queries (e.g.thrombectomy performed, tpa administered, etc.), and/or diagnostic labvalues

Full input is a concatenation of the above fields

Model:

XGBoost Model

Output:

Each episode across the entire EHR is labeled “high-risk of stroke” or“Not high-risk of stroke.” Patients may be further aggregated as a listof patient events with corresponding event dates from whichnotifications of high-risk status or indexing for clinical trials ordrug trials may be cultivated.

When ECG traces are selected from 9, 18, or 20 dB and EHR features areselected from age, gender, BMI, blood pressure, smoking status, LDL labresults, diagnosis of CHF, diagnosis of HTN, and diagnosis of diabetes,an exemplary composite model may perform at 70% or greater accuracy.

Inputs may be provided as raw values or categorical values. For acategorical model, the EHR Features may be split into Numeric Features(Age, BMI, BPs, LDL) which are merged with ECG predictions and thefeature value with measurement date closest to ECG test date may beselected as the input. For example, if the ECG date is July 15, andthere are dates of features captured on July 7th, July 16th, and July28th, then the selected date from which the input value is derived wouldbe July 16th because it is the closest measurement in time to the ECG.Features may be split off into Categorical Features (CHF, HTN, Diabetes,Smoker) based on whether each respective patient has the disease or not.Combined numerical and categorical patient feature models (with ECGdata) may perform at 83% or greater accuracy (Table 17).

Example 4 Results

TABLE 17 Model ROC AUC Sensitivity Specificity PPV NPV ECG Only 0.7310.725 0.620 0.073 0.983 EHR Only 0.829 0.818 0.691 0.094 0.990 ECG + EHR0.836 0.813 0.702 0.096 0.990 ECG + EHR + AF 0.843 0.802 0.726 0.1020.989

In one example, model inputs may include one or more patient featuresselected for their importance to the output high-risk label/prediction,including but not limited to demographic features such as age, sex, orother EHR-derived features, in addition to ECG-derived values. Forexample, LDL diagnostic testing values may account for the highestpercentage of the risk assessment and be a required input. In anotherexample, LDL values, ECGs, and age may collectively account for thehighest percentage of the risk assessment and be required inputs. In yetanother model, inputs may include LDL values, ECGs, age, blood pressure,HTN, AF status, BMI, diabetes, smoking, gender, and/or CHF. In an evenmore exhaustive implementation, a stroke prediction may include the topX features, where X is an integer. In one example, the integer, X, maybe 20 features. Consistent with a 20 feature embodiment, a top 20features may include: patient age, INDEX_CCI, STROKE_YN, AF_Target,Labs_AlC, Vitals_Weight, Vitals_Height, Demographics_SMOKER_FLG1,Anti_coag, Vitals_BMI, Labs_GLUCOSE, Labs_SODIUM, Vitals_BP_Systolic,Labs_LDL, Medications_ANTICOAGULANTS, Labs_HDL, Labs_HEMOGLOBIN,ECG_R_AXIS, and Echo_measurements_LAV_MOD_sp2.

In some embodiments, all features having a contribution to the high-riskdetermination with greater than a weight of 1% may be included in theinputs to a composite model. Consistent with a weight inclusion model,features may be selected to include: Vitals_BMI, Vitals_BP_Diastolic,Vitals_BP_Systolic, Vitals_Heart_Rate, Vitals_Height, Vitals_Weight,Demographics_FRS, Demographics_PCE, INDEX_CCI, CHADSVASC_SCORE,CHADS_SCORE, Demographics_PT_AGE, Demographics_PT_RACE,Demographics_PT_SEX, Demographics_SMOKER_FLG, ICD_Phenotypes_AOR,ICD_Phenotypes_AOS, ICD_Phenotypes_IVS, ICD_Phenotypes_LEF,ICD_Phenotypes_MIR, ICD_Phenotypes_MIS, ICD_Phenotypes_PUR,ICD_Phenotypes_PUS, ICD_Phenotypes_TRR, ICD_Phenotypes_TRS, Labs_AlC,Labs_BILI, Labs_BNP, Labs_BUN, Labs_CHOLESTEROL Labs_CKMB,Labs_CREATININE, Labs_CRP, Labs_D_dimer, Labs_eGFR, Labs_GLUCOSE,Labs_HDL, Labs_HEMOGLOBIN, Labs_LDH, Labs_LDL, Labs_LYMPHOCYTESLabs_POTASSIUM Labs_PRO_BNP, Labs_SODIUM, Labs_Triglyceride,Labs_TROPONIN_I, Labs_TROPONIN_T, Labs_URIC_ACID, Labs_VLDL,Medications_ACE_INHIBITORS,Medications_ANGIOTENSIN_II_RECEPTOR_ANTAGONISTS,Medications_ANTICOAGULANTS, Medications_ANTIDIABETIC_MEDICATION,Medications_ANTIHYPERTENSIVE, Medications_DIGOXIN,Medications_ERX_EBBB_HEART_FAILUREMedications_ERX_SPIRONOLACTONE_EPLE R,ONE_HEART_FAILURE, Medications_LOOP_DIURETICS, ECG_Measurements,Echo_Measurements, ECG_Findings, HF_YN, HTN_YN, AGE_GTE_75_YN, DM_YN,STROKE_YN, VASC_DISC_YN, AGE_65_74_YN, and FEMALE_YN.

Models based on the model inputs above may perform at 90% or greateraccuracy.

Example 5

In one embodiment, systems and methods described herein for predictionof atrial fibrillation from an ECG may further be adapted to predictother cardiac events from received ECG data. For example, of thereceived ECG data, measurements may record abnormal variations which aremeaningful in additional cardiac event analytics. The QT interval is onesuch measurement made on an ECG used to assess some of the electricalproperties of the heart. It is calculated as the time from the start ofthe Q wave to the end of the T wave, and approximates to the time takenfrom when the cardiac ventricles start to contract to when they finishrelaxing. An abnormally long or abnormally short QT interval isassociated with an increased risk of developing abnormal heart rhythmsand sudden cardiac death. Abnormalities in the QT interval can be causedby genetic conditions such as long QT syndrome, by certain medicationssuch as sotalol or pitolisant, by disturbances in the concentrations ofcertain salts within the blood such as hypokalaemia, by hormonalimbalances such as hypothyroidism, or they may be induced by certainmedications. QT prolongation is a measure of delayed ventricularrepolarization. Excessive QT prolongation can predispose the myocardiumto the development of early after-depolarisations, which in turn cantrigger re-entrant tachycardias such as torsades de pointes (TdP).Although the relationship between QT interval duration and the risk ofTdP is not fully understood, a corrected QT interval (QTc) of >500 ms oran increase in the QTc of >60 ms may be considered to confer a high riskof TdP in an individual patient. Prolongation of the corrected QT (QTc)interval becomes an even further concern, for example, with patients whoreceive psychotropic medications. Such patients may have baselineclinical risk factors for QTc prolongation, and many psychotropicmedications may further prolong this interval. Analytics may identifyover 200 medications having known or suspected association with QTcprolongation (LQT), which can lead to the rare but potentiallycatastrophic event, TdP.

Models herein generate predictions based upon the combination of ECGdata, patient age, and patient sex, although it will be appreciated thatother demographic features other than or in addition to one or both ofage or sex, and/or other EHR-derived features, may be used as modelinputs. Prediction of drug-induced LQT using an ECG-based machinelearning model is feasible and may outperform a model trained onbaseline QTc, age, and sex alone. In one example, ECG inputs having abaseline 12-lead ECGs with QTc values <500 ms for patients who had notreceived any known, conditional, or possible QTc prolonging medicationat the time of ECG or within the past 90 days may be matched with ECGsfrom the same patients while they were taking at least one drug(“on-drug” ECGs), such as one of the over 200 medications having knownor suspected associations with LQT. Features from the ECG as a whole maybe considered in addition to the presence of abnormal QTc features foreach respective patient.

Training may include using 5-fold cross-validation on a plurality ofmodels such as two machine learning models using the baseline ECGs ofapproximately 92,848 resulting pairs to predict drug-induced LQT (>500ms) in the on-drug ECGs. Artificial intelligence engines may beimplemented, including, by example, a deep neural network using ECGvoltage data and a gradient-boosted tree using the baseline QTc with ageand sex as additional inputs to both models. Other models may includeone or more inputs as described herein. Other combinations of folds,hold-out patients, validations, and number of models for comparison maybe considered without departing from the methodology as describedherein.

In one such training on an available patient dataset having paired ECGdata for patients with both an off-drug ECG and an on-drug ECG, on-drugLQT prevalence was 16%. The ECG model demonstrated superior performancein predicting on-drug LQT (area under the receiver operatingcharacteristic curve (AUC)=0.756) compared to the QTc model (0.710). Ata potential operating point such as depicted in FIG. 23 , the ECG modelhad 89% sensitivity and 95% negative predictive value. Even in thesubset of patients with baseline QTc <470/480 ms (male/female; post-drugLQT prevalence=14%), the ECG model demonstrated good performance(AUC=0.736). An ECG-based machine learning model can stratify patientsby risk of developing drug induced LQT better than a model usingbaseline QTc alone. This model may have clinical value to identifyhigh-risk drug starts that would benefit from closer monitoring andothers who are at low risk of drug induced LQT.

Patients having been identified as high risk for drug-induced LQT maythen be reported to their respective physicians for additionalmonitoring, potential therapy and treatment modifications, or otherrisk-reduction steps as determined by the physician. In one example, thereporting may include additional risk-reduction steps based upon one ormore personal characteristics of the patient, the patient's medicalhistory, the patient's ECG, or publications identified as beingpertinent to the patient based upon available data. In anotherembodiment, the high-risk identification may be generated real-time fromthe ECG equipment itself based upon the ongoing ECG and the patientcharacteristics uploaded to the equipment either manually by diagnosticpersonnel or retrieved from the patient's EMR linked to the ECGequipment.

Extension of Composite Model Implementations to Wearable Devices

Wearable devices, such as those having monitoring technology embedded inthe clothing or accessories of a subject, may include one or moremonitoring devices that capture instantaneous readings and/or readingsover time of heart rates, blood pressure, single or multiple probe ECG,temperature, presence and/or rate of perspiration, and other diagnosticmeasurements.

While one or more wearable devices are in use, a subject may bemonitored closely for incidences of active cardiac conditions/events orhigh risk of future cardiac conditions/events. For example, one or moreleads may be embedded in clothing of the subject which measure waveformssuch as those of corresponding ECG traces. In another example, awearable watch may include an ECG trace which measures the subject'srate and rhythm of heartbeats.

Each of the one or more wearable devices may provide monitoreddiagnostic information to a trained model which identifies the subject'srisk of a cardiac event. Upon detection of a high risk of present orfuture cardiac event, a notification may be provided to the subjectand/or their physician.

The trained model may reside in the wearable device, a subject's mobiledevice, a subject's desktop computer, a cloud-based system, or a remoteserver accessible through the internet or a local intranet.

Training the model may include one or more of the methodologiesdescribed herein with the addition of the instantaneous readings and/orreadings over time captured from the one or more wearable devices. Insome examples, the measurements taken from the wearable devices mayreplace one or more features of the inputs to the model such as thosefor measuring heart rate, heart rhythm, blood pressure, or one or moreECG or ECG-like leads.

For the purposes of training a model to predict high risk of a cardiacevent from the diagnostic data collected from a wearable device, a modelmay be trained using only diagnostic data collected from the wearabledevice, a model trained from the diagnostic data collected from awearable device may be improved or fine-tuned using additional dataoutside of the diagnostic data, or a model which has previously beentrained on a dataset may be translated to operate on the diagnostic datacollected from a wearable device.

For example, a model may be trained on ECG data such as a 12-leadelectrocardiogram (ECG) can include a I Lateral lead (also referred toas a I lead), a II Inferior lead (also referred to as a II lead), a IIIInferior lead (also referred to as a III lead), an aVR lead, an aVLLateral lead (also referred to as an aVL lead), an aVF Inferior lead(also referred to as an aVF lead), a V1 Septal lead (also referred to asa V1 lead), a V2 Septal lead (also referred to as a V2 lead), a V3Anterior lead (also referred to as a V3 lead), a V4 Anterior lead (alsoreferred to as a V4 lead), a V5 Lateral lead (also referred to as a V5lead), and a V6 Lateral lead (also referred to as a V6 lead).

Although the present disclosure discusses data ingestion from a 12-leadECG, it should be understood that it may be employed using data ingestedfrom ECGs with more or fewer leads, provided the ECG used for trainingrelies on a larger number of leads than the clinical or consumer devicethat is later used.

Similarly, although the portable or consumer devices to which thetrained model is applied are generally referred to herein as having onelead, it should be understood that they may include a larger number ofleads, provided that number is smaller than the number of leads on whichthe AI model is trained. For example, the device may be a single leaddevice such as a smart watch or other device worn on the wrist or adevice worn around the chest. Alternatively, the device may be amulti-lead device such as a garment with a pair of embedded leads. Stillfurther, the device may be a smaller, portable ECG device with, e.g., 1to 6 leads, or it may even be a clinical grade device, e.g., with 12leads. In the lattermost case, the device still preferably includesfewer leads than the device(s) from which the clinical data is obtained.

The present disclosure has applicability to multiple areas of medicinein which patient data is obtained via multi-lead ECGs. Such areas mayinclude, but are not limited to, cardiology, oncology, endocrinology,and medical diagnostics. Such areas may benefit from the transferlearning method disclosed herein due to the variability that isintroduced in data collection in each area. For example, in cardiology,oncology, and/or endocrinology, different machines will generatedifferent reads, so the transfer learning method makes it possible toevaluate this disparate data from one machine to the next, particularlywithout having to batch normalize the data. Similarly, with regard tomedical diagnostics, different labs may have their own procedures,biases, ranges for what is normal, etc. Additionally, data received fromone patient or cohort of patients may need to be modified in order torender it applicable to a second cohort of patients, e.g., data frommale patients and what is considered within normal ranges for them mayneed to be adjusted to apply it to female patients. Such modificationsalso may be accomplished through the transfer learning methods disclosedherein.

The present disclosure employs a transfer learning method to trainmillions of AI model parameters to predict a patient's current or futurehealth status with millions of 12-lead ECGs and paired clinical datafrom a healthcare provider. The method then includes taking the trained12-lead model, extracting interpretation units for individual leads, andthen reconstructing a model that will process data received from aclinical or consumer device that employs fewer leads. The method thenapplies a fine-tuning step in which the reconstructed model learns toadapt to the new device's data, where that step requires just a couplehundred samples (as opposed to the original millions).

Referring to FIG. 29 , the method 600 may include one or more steps ofdata ingestion, QA, or preprocessing 602. In particular, the method mayinclude time-series signal processing of ECG data and artifact detectionand exclusion. Ingestion may include, e.g., a plurality of voltage-timetraces where a first subset are stored at a first frequency, e.g., 500Hz, and a second subset are stored at a second, different frequency,e.g., 250 Hz. Such data may be batch loaded due to the exceedingly largevolume of clinical data being ingested, and similar batch techniques maybe applied to one or both of the training or prediction steps disclosedherein.

A preprocessing stage may include resampling the 250 Hz ECGs to 500 Hzby linear interpolation. Artifacts may include those identified by ECGsoftware at the time of ECG; for example, ECG outputs that include“technically limited”, “motion/baseline artifact”, “Warning:interpretation of this ECG, although attempted, may be adverselyaffected by data quality”, “Acquisition hardware fault prevents reliableanalysis”, “Suggest repeat tracing”, “chest leads probably not wellplaced”, “electrical/somatic/power line interference”, or “DefectiveECG”.

Pre-processing also may include identifying and excluding one or moresubsets of data. For example, when the model is designed to analyzeindividuals with respect to atrial fibrillation, a lead voltage over 12mV may be considered an exclusion criterion and/or considered to usuallyoccur as a result of motion artifacts. Thus, the method may perform aquality check in such instances and remove all ECG lead data at or abovethat threshold level. In another example, lead data reading 0 mV may beconsidered to result from a dead lead and may be deleted from thetraining set. Conversely, the method may retain such data for its model,recognizing that doing so may result in a dataset and model that aremore robust.

Pre-processing also may be applied to data received from the portable orconsumer device. For example, such devices may sample at a different,lower frequencies than clinical ECGs, so such data also may beprocessed, e.g., by linear interpolation, to adjust for the difference.

At step 604, the deep neural network parameters may be pretrained onmillions of 12-lead ECGs. This can involve just ECG data (unsupervised),or it may leverage associated clinical data (supervised). In someembodiments, the clinical data can include outcome data, such as whetheror not a patient developed AF in a time period following the day thatthe ECG was taken.

The method also may include mid-training network modification. Forexample, the network may be pruned and a single channel featurizationunit may be isolated. Such pruning may be useful to adapt the network tothe specific portable or consumer device being used. For example, forwrist-worn devices, the system may determine that a model trained andisolated on readings taken from I lead or II lead may be most similar ormost applicable. Alternatively, the system may determine that dataderived from a different lead or combination of leads may be mostapplicable for a chest-worn device that is placed over the wearer'sheart. One such pruning is done, new neural layers then may be added toconnect a single channel's features to a new classification layer.

Subsequently, at step 606, the method may resume training on a 1-channelECG dataset to fine-tune the model and then at step 608 apply andevaluate the model on data obtained from smaller-channel ECGs, e.g.,1-channel ECGs.

In some embodiments, the method may be diagnostic, whereby the clinicaldata can include outcome data, such as whether or not a patientdeveloped atrial fibrillation (AF or Afib) in a time period followingthe day that the ECG was taken. In other embodiments, the clinical datamay be used in a predictive sense, e.g., to determine based on that dataa likelihood that the patient would develop Afib within a certain timeperiod following the day that the ECG was taken.

AF is a cardiac rhythm disorder associated with several importantadverse health outcomes including stroke and heart failure. In patientswith AF and risk factors for thromboembolism, early anticoagulation hasbeen shown to be effective at preventing strokes. Unfortunately, AFoften goes unrecognized and untreated since it is frequentlyasymptomatic or minimally symptomatic. Thus, systems and methods toscreen for and identify undetected AF can assist in preventing strokes.

Population-based screening for AF is challenging for two primaryreasons. One, the yearly incidence of AF in the general population islow with reported incidence rates of less than 10 per 1000 person yearsunder the age of 70. Two, AF is often “paroxysmal” (the patient goes inand out of AF for periods of time) with many episodes lasting less than24 hours. Currently, the most common screening strategy is opportunisticpulse palpation, sometimes in conjunction with a 12-leadelectrocardiogram during routine medical visits. This has been shown tobe cost-effective in certain populations and is recommended in someguidelines. However, studies of implantable cardiac devices havesuggested that this strategy will miss many cases of AF.

A number of continuous monitoring devices are now available to detectparoxysmal and asymptomatic AF. Patch monitors can be worn for up to14-30 days, implantable loop recorders provide continuous monitoring foras long as 3 years, and wearable monitors, sometimes used in conjunctionwith mobile devices, can be worn indefinitely. Continuous monitoringdevices overcome the problem of paroxysmal AF but must still contendwith the overall low incidence of new onset AF and cost and conveniencelimit their use for widespread population screening.

FIGS. 30 and 31 provide model performance metrics for 1 yr first timeincident Afib risk towards patients aged >=18 years. In both cases,Mann-Whitney U tests with Bonferroni corrections were used to assesssignificant differences between groups. “*****” indicates statisticallysignificant with a p-value <0.05, “ns” indicates that the differencebetween groups was not statistically significant, and “**” indicatessome statistical significance.

In particular, model performance in FIG. 30 is depicted using receiveroperating characteristic area under the curve (ROC AUC). ROC AUC is arobust metric of model performance that represents the ability todiscriminate between two classes. Higher ROC AUC suggests higherperformance (with perfect discrimination represented by an ROC AUC of 1and an AUROC of 0.5 being equivalent to a random guess).

Model performance in FIG. 31 is depicted using precision recall areaunder the curve (PR AUC). PR AUC is an average precision scoredetermined by computing weighted average of precisions achieved at eachthreshold by the increase in recall.

The ROC AUC and PRC AUC of the model for the prediction of new onset AFwithin 1 year were approximately 0.828, 95% CI [0.827, 0.829] and 0.194[0.192, 0.197], respectively, for Lead I, 0.832 [0.831, 0.833] and 0.207[0.205, 0.209], respectively, for Leads I and II, 0.833 [0.0832, 0.835]and 0.207 [0.205, 0.210], respectively, for Leads V1-V6, and 0.834[0.833, 0.836] and 0.210 [0.209, 0.211], respectively, for Leads I, II,and V1-V6.

These results demonstrate that the AI model may be properly trained onclinical data and then applied to data received from portable orconsumer devices, permitting the use of cardiology analysis outside of aclinical setting.

FIGS. 4A and 4B are exemplary embodiments of models usable with themethod disclosed herein. The disclosure provided above with respect tothose models also is applicable to the examples and methods disclosedherein.

In one embodiment, the convolutional neural networks, such as thosedepicted in FIGS. 4A and 4B, may be trained on a first set of data andtranslated to perform on a second set of data. In one example, the firstset of data may be robust and include large quantities of samples fromwhich to train while the second set of data may be sparse and includeonly a few quantities of samples. In another example, the first set ofdata may include more operational parameters or features, such as havingaccess to more clinical data or more complete diagnostic data.Diagnostic data may include those from different disease states such asoncology, cardiology, endocrinology, and diagnostic laboratory testing.In the field of oncology, for example, a first dataset may include thefull RNA transcriptome and subsequent read quantities generated fromnext generation sequencing while the second set of data may have beengenerated from a greatly reduced number of transcriptomes such as thosegenerated from a smaller panel or microarray.

In another example, a model may be translated from training from onesequencing laboratory to another sequencing laboratory due to thedifferences in the laboratories' equipment or sequencing procedures. Insuch an example, the datasets may not be categorized as a robust tosparse but instead as robust to robust, but where there exists disparitybetween the data. One aspect of translating a model trained on a robustdataset to another robust dataset is that the model maintainsperformance while becoming generalizable across many different robustdatasets without concern for where they were generated.

Multiple embodiments may be implemented on data varying in quantity,quality, and number of features. In general, a model may be trained on adataset having high quantity of samples, higher quality of samples,and/or higher number of features for each sample, and may be translatedto a dataset having a lower quantity of samples, lower quality ofsamples, and/or a lower number of features for each sample. In thismanner, higher quality predictive algorithms may be adapted forperformance on datasets having one or more disadvantages that precludethe dataset from being used to generate a higher quality model.

Similar to RNA as presented above a model trained on a first DNA panelmay be translated using a second DNA panel in whether a robust to sparsetranslation or a robust to robust translation.

If a user attempted to plug the reduced data set into a model trainedfrom the robust dataset, the results generated would not be accurate dueto the differences in data. However, if the model was translated fromthe first dataset to the second dataset, much of the performance of themodel generated from the robust training set of data may be retained foruse with the reduced set of data.

In the field of endocrinology, a similar translation may be performed,for example, on sequencing data generated for treating a patient havingdiabetes, or other endocrinological diagnosis.

In the field of mental health, a similar translation may be performed,for example, on sequencing data generated for treating a patient havingdepression, or other mental health diagnosis.

In the field of laboratory testing, a similar translation may beperformed, for example, on diagnostic laboratory tests for metabolicpanels, blood panels, viral or bacterial panels, or other laboratorydiagnostic testing.

Steps for translating the model may include performing one or moretransfer learning methodologies.

In the field of cardiology, the robust dataset may include 12 lead ECGsacross millions of patients while the reduced dataset may be limited toa few leads such as those generated from one or more wearable devices.In some embodiments, the translation may be performed across differentECG collection devices, whether having the same number of leads in arobust to robust translation, a differing number of leads in a robust torobust translation, a same number of leads in a robust to sparsetranslation, or a differing number of leads a robust to sparsetranslation, where robust may refer to the number of samples in thedataset, the quality of samples in the dataset, or the number offeatures associated with the samples of the dataset.

Before training, a time-series signal processing of ECG data includingartifact detection and exclusion may be performed. This includespreprocessing steps such as sampling normalization, voltage tracestructure changes, and possible inclusions of noisy data to regularizedeep learning models. For example, dead leads and/or spikes inmillivolts may be identified (such as over 12 mv).

The deep neural network parameters may be pretrained on millions of12-lead ECGs. This can involve just ECG data (unsupervised), or it mayleverage associated clinical data such as patient demographics,diagnoses, or cardiac anatomy and functional measures (blood flow fromheart) (supervised). In some embodiments, the clinical data can includeoutcome data, such as whether or not a patient developed AF in a timeperiod following the day that the ECG was taken. The resulting neuralnetwork may be composed of model specific convolutional layer blocks,and/or fully connected layers, such as those presented in exemplaryarchitectures 9 a and 9 b.

The method also may include mid-training network modification. Forexample, the network may be pruned and a single channel featurizationunit may be isolated. Such pruning may be useful to adapt the network tothe specific portable or consumer device being used. For example, forwrist-worn devices, the system may determine that a model trained andisolated on readings taken from I lead or II lead may be most similar ormost applicable. Alternatively, the system may determine that dataderived from a different lead or combination of leads may be mostapplicable for a chest-worn device that is placed over the wearer'sheart. By identifying a corresponding lead within the trainedconvolutional network, it may be held out from pruning or selected forpruning based on the desire to include or exclude it from the translatedmodel. One such pruning is done, new neural layers then may be added toconnect a single channel's features to a new classification layer.

In one embodiment, the frozen layers may be the GAP layers of FIGS. 4Aand 4B. In another embodiment, the frozen layers may be the dense layersof FIGS. 4A and 4B. In yet another embodiment, one or more of the GAPlayers may be selected for freezing and or other layers as identifiedusing the rule set or heuristic algorithms.

In another example, pruning the network and extracting a subset (1 to12) of the lead featurization units may be performed via a derivedinsights table with pre-programmed rules, or in a programmatic mannerusing one or optimization or heuristic models before adding new neurallayers to connect channel features to a new classification layer.

Subsequently the translation steps may resume training on a 1-channelECG dataset to fine-tune the model and before being able to apply andevaluate the transformed model on data obtained from smaller-channelECGs, e.g., 1-channel ECGs. Fine tuning may include training on adataset that matches the pruned input structure. Fine-tuning strategiescan either freeze the extracted ECG layers and retrain the unfrozenfinal layers, or “un-freezing” the ECG layers to further modify thefeaturization of ECG leads. Modifying which layers are exempt fromretraining at each fine-tuning iteration enables the model to select forthe best layers to reweigh and improve the resulting translated model.

In one embodiment, the frozen layers may be the GAP layers of FIGS. 4Aand 4B. In another embodiment, the frozen layers may be the dense layersof FIGS. 4A and 4B. In yet another embodiment, one or more of the GAPlayers may be selected for freezing and or other layers as identifiedusing the rule set or heuristic algorithms.

In some embodiments, the method may be diagnostic, whereby the clinicaldata can include outcome data, such as whether or not a patientdeveloped atrial fibrillation (AF or Afib) in a time period followingthe day that the ECG was taken. In other embodiments, the clinical datamay be used in a predictive sense, e.g., to determine based on that dataa likelihood that the patient would develop Afib within a certain timeperiod following the day that the ECG was taken.

While examples provided herein include one or more combinations of modelinputs, exemplary combinations of model inputs may be selected from anypatient features within the EMR.

While the invention may be susceptible to various modifications andalternative forms, specific embodiments have been shown by way ofexample in the drawings and have been described in detail herein.However, it should be understood that the invention is not intended tobe limited to the particular forms disclosed.

Thus, the invention is to coverall modifications, equivalents, andalternatives falling within the spirit and scope of the invention asdefined by the following appended claims.

To apprise the public of the scope of this invention, the followingclaims are made:

What is claimed is:
 1. A method for determining cardiac disease riskfrom electrocardiogram trace data, comprising: receivingelectrocardiogram trace data associated with a patient, theelectrocardiogram trace data having an electrocardiogram configurationincluding a plurality of leads and a time interval and comprising, foreach lead included in the plurality of leads, voltage data associatedwith at least a portion of the time interval; identifying one or moreleads of the plurality of leads of the electrocardiogram trace data, theone or more leads derivable from a combination of other leads of theplurality of leads, wherein a portion of the electrocardiogram tracedata does not include electrocardiogram trace data of the one or moreleads; providing the portion of the electrocardiogram trace data to atrained machine learning model, the model trained to evaluate theportion of the electrocardiogram trace data with respect to one or morecardiac disease states; and generating, by the trained machine learningmodel and based on the evaluation, a risk score reflecting a likelihoodof the patient being diagnosed with a cardiac disease state within apredetermined period of time from when the electrocardiogram trace datawas generated; and outputting the risk score to at least one of a memoryor a display.
 2. The method of claim 1, further comprising: generating,based on the risk score and an additional risk score associated with asecond cardiac disease state, a composite prediction, the compositeprediction reflecting a likelihood that the patient will be diagnosedwith either of the cardiac disease state or the second cardiac diseasestate within a predetermined period of time from when theelectrocardiogram data was generated.
 3. The method of claim 1, whereinthe electrocardiogram trace data does not include features indicatingthat the patient has the cardiac disease state when theelectrocardiogram trace data was generated.
 4. The method of claim 1,further comprising generating, based on the risk score, a treatmentrecommendation; and outputting the treatment recommendation to at leastone of the output or the display, wherein the treatment recommendationincludes a recommendation to perform additional cardiac monitoring forthe patient.
 5. The method of claim 1, wherein the trained model furthercomprises: training a convolutional neural network on a first pluralityof patients and a second plurality of patients, wherein a time between adate of a recorded ECG and a diagnosis of aortic stenosis for each ofthe first plurality of patients is less than a diagnosis threshold, anda time between a date of a recorded ECG and a diagnosis of aorticstenosis for each of the second plurality of patients is greater thanthe diagnosis threshold; wherein a time between a date of a recorded ECGand a diagnosis of aortic stenosis that is less than the diagnosisthreshold is indicative of an active aortic stenosis at the date of therecorded ECG; and providing the trained convolutional neural network asthe trained model.
 6. The method of claim 5, wherein the trained modelfurther comprises: refining the trained neural network using only thesecond plurality of patients, and wherein the diagnosis threshold isselected from a number of days.
 7. The method of claim 1, wherein thetrained model is selected based at least in part on a severity of thecardiac disease state the generated risk score represents.
 8. The methodof claim 1, wherein the trained model comprises training data associatedwith a plurality of clinical sites.
 9. The method of claim 8, whereinthe trained model comprises training the model using patient dataassociated with one site of the plurality of sites and testing thetrained model on the remaining sites of the plurality of sites.
 10. Themethod of claim 1, wherein the electrocardiogram trace data comprisesECG data selected from one or more of acute myocardial infarction,atrial fibrillation, atrial flutter, complete block, earlyrepolarization, fascicular block, first-degree atrioventricular block,intraventricular conduction block, left bundle branch block, rightbundle branch block, ischemia, left anterior descending artery ischemia,right bundle branch block, low QRS, left ventricular hypertrophy,non-specific ST-T wave, Non-specific T wave, other bradycardia,premature atrial contractions, pacemaker, poor tracing, priorinfarction, prior myocardial infarction anterior, prolonged QT,premature ventricular contractions, right axis deviation, second degreeatrioventricular block, sinus bradycardia, supraventricular tachycardia,tachycardia, tachyarrhythmia, T inversion, or ventricular tachycardia.11. The method of claim 1, wherein the electrocardiogram trace datacomprises at least 8 leads.
 12. The method of claim 1, wherein theelectrocardiogram trace data is sampled at 250 hz or 500 hz.
 13. Themethod of claim 1, wherein the trained model is a composite model, andwherein training the trained composite model comprises: generating apatient timeline for each patient; anchoring each respective patienttimeline to a date of occurrence of an echocardiogram; and labeling eachrespective patient as having a positive or negative ECG based at leastin part on a date of an ECG with respect to the date of occurrence ofthe echocardiogram.
 14. The method of claim 13, further comprisesexcluding patients from training after a censoring event is detected inthe patient timeline.
 15. The method of claim 1, wherein the cardiacdisease state is one of aortic stenosis, aortic regurgitation, mitralstenosis, mitral regurgitation, tricuspid regurgitation, reducedejection fraction, increased interventricular septal thickness, cardiacamyloidosis, or stroke.
 16. The method of claim 1, wherein the trainedmachine learning model is trained to evaluate patient data of anelectronic health record with respect to one or more cardiac diseasestates.
 17. A system comprising: a computer including a processingdevice, the processing device configured to: receive electrocardiogramtrace data associated with a patient, the electrocardiogram trace datahaving an electrocardiogram configuration including a plurality of leadsand a time interval and comprising, for each lead included in theplurality of leads, voltage data associated with at least a portion ofthe time interval; identify one or more leads of the plurality of leadsof the electrocardiogram trace data, the one or more leads derivablefrom a combination of other leads of the plurality of leads, wherein aportion of the electrocardiogram trace data does not includeelectrocardiogram trace data of the one or more leads; provide theportion of the electrocardiogram trace data to a trained machinelearning model, the model trained to evaluate the portion of theelectrocardiogram trace data with respect to one or more cardiac diseasestates; generate, by the trained machine learning model and based on theevaluation, a risk score reflecting a likelihood of the patient beingdiagnosed with a cardiac disease state within a predetermined period oftime from when the electrocardiogram trace data was generated; andoutput the risk score to at least one of a memory or a display.
 18. Thesystem of claim 17, wherein the cardiac disease state is one of aorticstenosis, aortic regurgitation, mitral stenosis, mitral regurgitation,tricuspid regurgitation, reduced ejection fraction, increasedinterventricular septal thickness, cardiac amyloidosis, or stroke.
 19. Anon-transitory computer readable medium, comprising instructions forcausing a computer to: receive electrocardiogram trace data associatedwith a patient, the electrocardiogram trace data having anelectrocardiogram configuration including a plurality of leads and atime interval and comprising, for each lead included in the plurality ofleads, voltage data associated with at least a portion of the timeinterval; identify one or more leads of the plurality of leads of theelectrocardiogram trace data, the one or more leads derivable from acombination of other leads of the plurality of leads, wherein a portionof the electrocardiogram trace data does not include electrocardiogramtrace data of the one or more leads; provide the portion of theelectrocardiogram trace data to a trained machine learning model, themodel trained to evaluate the portion of the electrocardiogram tracedata with respect to one or more cardiac disease states; generate, bythe trained machine learning model and based on the evaluation, a riskscore reflecting a likelihood of the patient being diagnosed with acardiac disease state within a predetermined period of time from whenthe electrocardiogram trace data was generated; and output the riskscore to at least one of a memory or a display.
 20. The non-transitorycomputer readable medium of claim 19, wherein the cardiac disease stateis one of aortic stenosis, aortic regurgitation, mitral stenosis, mitralregurgitation, tricuspid regurgitation, reduced ejection fraction,increased interventricular septal thickness, cardiac amyloidosis, orstroke.